version: 2
snapshot:
widths: [1024, 768, 375]
min-height: 1024
percy-css: >
.skeleton { animation: none !important; }
[data-testid="loading"] { visibility: hidden !important; }
Components are organized by route: /issues , /diff , /
settings , /history . Each route is rendered with a seeded fixture
to ensure deterministic output.
Baseline Image Generation and Storage
Baselines are stored in Percy's cloud and mirrored to an S3 bucket
for disaster recovery. Baseline promotion follows a two-step process:
1. Draft: PRs generate draft snapshots. Only the PR author and
designated reviewers can approve baselines.
2. Approved: Once approved, baselines are locked. Future
changes that exceed the diff threshold (0.1% pixel difference)
fail the CI job.
Baselines are tagged with upcodeVersion and gitSha . A nightly
job cleans baselines older than 90 days unless tagged keep .
Cross-Browser Visual Diff Analysis
Percy captures screenshots in Chrome, Firefox, and Safari (via
BrowserStack). Cross-browser diffs are computed separately:
# Percy cross-browser diff
percy exec -- chromium firefox webkit -- pnpm test:visual
# Output:
# Chrome vs Firefox: 0.02% diff (within threshold)
# Chrome vs Safari: 0.15% diff (exceeds threshold - review required)
BrowserStack credentials are stored in GitHub Secrets. Snapshots
are rotated to prevent caching artifacts.
Responsive Breakpoint Coverage
All components are tested at three breakpoints:
Breakpoint Width Device
Desktop 1024px+ Laptop / external monitor
Tablet 768px iPad
Mobile 375px iPhone SE
The responsive test suite runs in a matrix job with three parallel
Playwright workers. Each worker sets the viewport size before
rendering. Layout shifts are detected via CLS (Cumulative Layout
Shift) measurement and must be below 0.1.
Theme Variant Visual Testing
Each component is tested against the four VS Code built-in themes:
Light+, Dark+, High Contrast Light, High Contrast Dark. Custom
theme previews are generated from the design token set and
compared against reference images.
Theme switching is simulated by injecting the theme's CSS variable
set before rendering. High contrast themes are validated for forced-
colors compatibility using the Windows high contrast emulator in
Playwright.
CI Integration and Approval Workflow
Visual tests run on every PR via a Percy GitHub Action. The job gates
on percy/upload success and percy/diff status.
- name: Visual Regression Tests
uses: percy/exec-action@v0.4.1
with:
command: "pnpm test:visual"
env:
PERCY_TOKEN: ${{ secrets.PERCY_TOKEN }}
If diffs are within the auto-approve threshold (0.05%), the job
passes automatically. Otherwise, a Percy review is posted on the PR.
The PR cannot be merged until a reviewer clicks "Approve" in the
Percy dashboard.
Contract Testing: Consumer-Driven
Contracts
Pact Contract Definitions per Integration Point
Pact contracts are defined in pacts/ with one file per consumer-
provider pair. Contracts specify the expected request shape and
response for each integration point:
{
"consumer": { "name": "upcode-webview" },
"provider": { "name": "upcode-extension-host" },
"interactions": [
{
"description": "request scan results for a file",
"request": {
"method": "POST",
"path": "/scan",
"headers": { "Content-Type": "application/json" },
"body": { "filePath": "/src/[Link]", "ruleset": "recommended
},
"response": {
"status": 200,
"headers": { "Content-Type": "application/json" },
"body": {
"issues": [
{ "ruleId": "js/no-var/v2", "severity": "warning", "line":
]
}
}
}
]
}
Contracts are versioned in Git. Provider verification runs against the
latest consumer contract on every provider build.
Contract Verification in CI
Each provider runs a Pact verification step in its CI pipeline.
Verification results are published to the Pact Broker:
# Provider verification
pnpm pact:verify --provider-url [Link] \
--pact-urls ./pacts/[Link] \
--provider-states-setup-url [Link]
Verification failures block the provider build. If a consumer contract is
incompatible with the current provider implementation, the CI job
fails with a PACT_MISMATCH error and a diff of the expected vs
actual response.
Drift Detection and Alerting
The Pact Broker is queried nightly for contract drift. Drift is defined
as: a provider's latest version is not referenced by any consumer's
latest contract. Drift alerts are sent to #pact-alerts in Slack.
# Can i deploy - provider
pact-broker can-i-deploy --pacticipant upcode-extension-host \
--version v2.1.0 --to-environment production
If the provider has breaking changes not yet reflected in consumer
contracts, the command exits with a non-zero code and CI fails.
Breaking Change Prevention
Semantic versioning is enforced at the contract level. Any provider
change that modifies a required field, removes an endpoint, or
changes a response type must bump the major version. The CI
pipeline runs pact-broker diff between the current and previous
provider versions:
pact-broker diff --pacticipant upcode-extension-host \
--versions v2.0.0..v2.1.0
Breaking changes require a migration plan documented in pacts/
[Link] . Consumer teams are notified via GitHub issue
created by a bot.
Mock Service Configuration
Pact mock services run in Docker during local development and CI.
Configuration is centralized in pacts/[Link] :
[Link] = {
mockService: {
port: 1234,
host: 'localhost',
ssl: false,
},
broker: {
url: '[Link]
username: [Link].PACT_BROKER_USERNAME,
password: [Link].PACT_BROKER_PASSWORD,
},
};
Consumers use the mock service in integration tests. The mock
service is reset between test suites to ensure isolation.
Chaos Testing and Fault Injection
Parser Failure Injection
The parser layer is wrapped in a fault injector that simulates crashes
at configurable rates. The injector is enabled only in test
environments via the CHAOS_PARSER_FAILURE_RATE environment
variable:
function parseWithChaos(source: string, options: ParserOptions) {
if ([Link].CHAOS_PARSER_FAILURE_RATE && [Link]() < parseFl
throw new Error('Chaos: simulated parser crash');
}
return parseJavaScript(source, options);
}
Chaos tests verify that parser failures do not crash the extension
host and that the error is surfaced to the user as a skipped file with a
descriptive message.
Network Partition Simulation
Cloud-mode operations are tested against simulated network
partitions using toxiproxy . A toxiproxy instance is started in
Docker and configured to drop all packets between the extension
host and the cloud API:
# Start toxiproxy
docker run -d -p 8474:8474 -p 8666:8666 shopify/toxiproxy
# Add a toxic that drops 100% of packets
curl -X POST [Link] \
-d '{"type": "timeout", "attributes": {"timeout": 0}}'
Tests verify that the extension falls back to offline mode, queues
mutations, and syncs when connectivity is restored.
Disk Full Simulation
File write operations are tested against a full disk by mounting a
tmpfs filesystem, filling it to capacity, and attempting to write
backup files:
# Create 1GB tmpfs and fill it
mount -t tmpfs -o size=1g tmpfs /tmp/upcode-test
dd if=/dev/zero of=/tmp/upcode-test/fill bs=1M count=1024
# Attempt backup write
pnpm test:backup -- --target /tmp/upcode-test
Expected behavior: backup is skipped, an error is reported, and the
extension continues operating without crashing.
OOM Simulation
Memory-heavy operations (scanning large workspaces) are tested
under artificial memory pressure using stress-ng on Linux and
memory_pressure on macOS:
# Linux: stress 2GB of memory
stress-ng --vm 2 --vm-bytes 2G --timeout 60
# Run large workspace scan in parallel
pnpm test:stress -- --workspace ./fixtures/large-workspace
Tests verify that the extension process is OOM-killed gracefully (exit
code 137) and that VS Code shows a recoverable error rather than
crashing the entire window.
Graceful Degradation Verification
When any subsystem fails (parser, cloud, transformer), the extension
degrades gracefully:
• Parser fails: File is skipped; issue count reflects only parseable
files.
• Cloud unreachable: Local analysis continues; mutations are
queued for later sync.
• Transformer crashes: Issue is marked as "fix unavailable";
other fixes continue.
• Settings sync fails: Local settings are used; sync is retried on
next activation.
Chaos tests verify each degradation path individually and in
combination.
Recovery Time Measurement
Recovery time is measured from the moment a fault is cleared to the
moment the extension returns to fully operational state.
Measurements are recorded in CI and compared against SLOs:
Recovery
Subsystem Measurement Method
SLO
Time from injector off to next
Parser < 100ms
successful parse
Time from toxiproxy disable to cloud
Network < 5s
API response
Time from disk freed to next
Disk < 500ms
successful backup write
Time from OOM kill to extension re-
Memory < 2s
activation
Recovery times exceeding SLOs trigger a CI warning. Patterns that
consistently exceed SLOs require architectural review.
Performance Testing Deep Dive
Benchmark Harness Design
The benchmark harness runs rule-level performance tests against a
fixed corpus. The corpus is versioned and stored in benchmarks/
corpus/ :
interface BenchmarkSuite {
name: string;
corpus: BenchmarkFile[];
iterations: number;
warmup: boolean;
}
interface BenchmarkFile {
path: string;
language: Language;
sizeKB: number;
content: string;
}
const suites: BenchmarkSuite[] = [
{
name: 'parser-baseline',
corpus: loadCorpus('benchmarks/corpus/js/100kb/'),
iterations: 100,
warmup: true,
},
{
name: 'rule-throughput',
corpus: loadCorpus('benchmarks/corpus/mixed/10mb/'),
iterations: 10,
warmup: false,
},
];
The harness uses benchmark (from benchmarkjs ) to compute
statistics. Results are output as JSON for CI consumption.
Automated Regression Detection
Benchmark results are compared against the baseline stored in
performance/[Link] . The comparison uses a
two-tailed Mann-Whitney U test (non-parametric) to detect shifts in
distribution:
function detectRegression(
baseline: number[],
current: number[]
): { isRegression: boolean; pValue: number; percentChange: number } {
const u = mannWhitneyUTest(baseline, current);
const percentChange = (median(current) - median(baseline)) / median(
return {
isRegression: [Link] < 0.01 && percentChange > 0.1,
pValue: [Link],
percentChange,
};
}
Regressions are posted as GitHub PR comments with a flame graph
diff. Teams can override with /perf-allow <suite> <duration> .
Memory Profiling in CI
Memory profiling runs on the dedicated benchmark-mac runner.
Heap snapshots are taken before and after each benchmark suite
using node --heap-prof :
node --heap-prof --heap-prof-dir ./heap-profiles \
--expose-gc \
benchmarks/[Link]