Component Testing at 1M Runs/Month: Playwright CT vs Cypress CT vs Vitest Browser Mode

Most comparison articles benchmark three tools on a laptop and report a stopwatch number. That is useless for an organisation running a million component test executions a month, because the decision variables are not throughput — they are p95 PR feedback latency, first-attempt flake rate, and the fully loaded cost of a failing run in engineer-minutes.

This article reports a controlled benchmark of all three runners against an identical 10,000-test React suite, then derives a cost model you can re-parameterise with your own numbers.

The Benchmark Contract: What "1M Runs" Actually Means

Ambiguity kills credibility, so define the unit first.

• One run = one test case execution (one test() / it() body completing).
• Reference workload = 10,000 component test cases, executed ~100 times per month across PR lanes, merge queues, and nightly jobs → 1,000,000 runs/month.
• Sharding = 8 shards, each a 4 vCPU runner with workers: 4.

The measurement rig

Pin your versions before trusting any number below. Playwright CT and Vitest browser mode have both shipped breaking changes inside minor releases, and Cypress's Vite-based CT bundler has materially different cold-start behaviour than its webpack path.

How flake rate was isolated

A failing run is only a flake if it cannot be reproduced. Our protocol: every failing run is re-executed three times on the identical commit SHA, lockfile, and browser version in an isolated runner. Failure on 0 of 3 retries = flake. Failure on 1 or more = real defect. This is the only methodology that separates runner instability from your own race conditions.

p95 Wall Time: Where the Milliseconds Actually Go

Why Cypress carries the heaviest tail

Cypress CT compiles each spec through its dev server, boots the Cypress runner application, injects your component under test into an iframe, and then routes every cy.* command across the runner↔AUT bridge. Each command owns a retry-and-timeout loop that is serialised through a single command queue. That design is a genuine debugging superpower — you get a time-travel snapshot of every step — but it is structurally hostile to tail latency, because one slow assertion blocks everything behind it in the queue.

Why Playwright CT's cost moves to the front

Playwright CT uses Vite to produce an instrumented bundle before the browser launches, then drives a real page through the devtools protocol. Assertions auto-wait in-page rather than round-tripping through a host application, which is why its per-test p95 lands 30% under Cypress. The trade: you pay a ~2× larger shard cold start, which only hurts when shards are short — a real concern at high parallelism.

Why Vitest browser mode wins on both axes

Vitest browser mode serves your tests as native ESM from the Vite dev server and executes them inside the browser. The Node-side orchestrator is a thin WebSocket client collecting results. There is no bundling step for the test itself, and the module graph is reused across spec files within a worker — so both cold start and marginal per-test cost collapse.

The p95/throughput distinction nobody states clearly

These are different questions and they have different answers:

• p95 is a developer-experience metric. It is what your engineer experiences on a bad day, and it drives context-switching.
• Throughput is a finance metric. It is what appears on the invoice.

In a closed-loop system, tail latency does not dominate throughput until utilisation crosses roughly 70% (queueing theory: you pay for the tail when demand approaches capacity). Benchmark harnesses that report only one number are hiding the trade-off. See Dean & Barroso's The Tail at Scale for the canonical treatment.

Operational rule: never raise workers above your vCPU count hoping for throughput. Above that line you buy tail latency and CPU contention, not capacity — our 4 vCPU / 4 worker config already oversubscribes when each Chromium context spawns its own renderer and GPU process.

Flake Rate: The Metric That Actually Kills Teams

Over 500,000 first-attempt executions per runner, flake rates were:

These gaps are not primarily a property of the runners. Root-cause attribution across 1,842 triaged flakes showed the following distribution — and roughly 60% of it is fixable in your own test code regardless of runner.

The retry trap

Setting retries: 2 in your component lane does not reduce flake. It converts a measurement problem into a hidden tax: your dashboard reports 0.08% while your engineers actually experience 1.3%, and every masked failure has already burned a queue slot and 90 seconds of wall clock.

Run retries: 0 in the PR lane and isolate flake detection in a separate nightly job that reruns the same SHA three times.

Determinism discipline that removes 60% of flakes

// vitest.setup.ts — shared state lifecycle is the single highest-leverage fix
import { afterAll, afterEach, beforeAll } from 'vitest';
import { cleanup } from '@testing-library/react';
import { setupServer } from 'msw/node';

export const server = setupServer();

beforeAll(() => {
  server.listen({
    // Fail loudly on unmocked calls instead of silently hitting the network.
    onUnhandledRequest: 'error',
  });
});

afterEach(() => {
  server.resetHandlers(); // drop per-test overrides, keep base handlers
  cleanup();              // unmount React trees + reset the DOM
  localStorage.clear();
  sessionStorage.clear();
});

afterAll(() => server.close());

The True Cost Model: Compute Is a Rounding Error

The invoice: CI minutes

At a fully loaded self-hosted rate of $0.042/runner-minute (raw spot cost plus control plane, autoscaling headroom, and SRE time — not the headline GitHub-hosted rate, which excludes idle capacity):

Cypress CT's invoice is 2.4× Vitest's. Meaningful, but not existential.

The hidden 98%: flake triage

Now add humans. Assumptions, all tunable: a blended loaded engineer rate of $110/hr ($1.833/min), 35% of flakes investigated (the rest get blind-rerun), and 6 minutes of average triage per investigated flake including the context switch back.

Compute is 0.48%, 0.37%, and 0.50% of the bill respectively. Triage is ~98%.

The strategic conclusion writes itself: optimising CI minutes is theatre. Cutting your first-attempt flake rate from 1.3% to 0.4% is worth roughly $35,000 per million runs, and it compounds as your suite grows.

The model, in code

Make your assumptions auditable — this is the thing you paste into the architecture decision record.

type Usd = number;
type Minutes = number;

export interface RunnerProfile {
  readonly id: 'playwright-ct' | 'cypress-ct' | 'vitest-browser';
  readonly testsPerSecondPerRunner: number;
  readonly shardOverheadSeconds: number;
  readonly costPerRunnerMinuteUsd: Usd;
}

export interface OrgProfile {
  readonly runsPerMonth: number;
  readonly testCasesPerSuite: number;
  readonly shards: number;
  readonly firstAttemptFlakeRate: number; // 0..1
  readonly rerunsPerFlake: number;
  readonly investigatedFlakeShare: number; // 0..1
  readonly triageMinutes: Minutes;
  readonly engineerUsdPerMinute: Usd;
  readonly blockedShareWhileWaiting: number; // 0..1
}

export interface CostBreakdown {
  readonly computeUsd: Usd;
  readonly retryComputeUsd: Usd;
  readonly developerWaitUsd: Usd;
  readonly flakeTriageUsd: Usd;
  readonly totalUsd: Usd;
  readonly p95SuiteSeconds: number;
  readonly flakeEvents: number;
}

function invariant(condition: unknown, message: string): asserts condition {
  if (!condition) throw new RangeError(`[ci-cost-model] ${message}`);
}

export function modelCost(
  runner: RunnerProfile,
  org: OrgProfile,
  p95ToP50Ratio = 1.3,
): CostBreakdown {
  invariant(runner.testsPerSecondPerRunner > 0, 'testsPerSecondPerRunner must be > 0');
  invariant(runner.costPerRunnerMinuteUsd > 0, 'costPerRunnerMinuteUsd must be > 0');
  invariant(org.runsPerMonth > 0, 'runsPerMonth must be > 0');
  invariant(org.testCasesPerSuite > 0, 'testCasesPerSuite must be > 0');
  invariant(Number.isInteger(org.shards) && org.shards > 0, 'shards must be a positive integer');
  invariant(
    org.firstAttemptFlakeRate >= 0 && org.firstAttemptFlakeRate < 1,
    'firstAttemptFlakeRate must be in [0, 1)',
  );
  invariant(
    org.investigatedFlakeShare >= 0 && org.investigatedFlakeShare <= 1,
    'investigatedFlakeShare must be in [0, 1]',
  );

  const suiteRuns = org.runsPerMonth / org.testCasesPerSuite;

  const testSeconds = org.runsPerMonth / runner.testsPerSecondPerRunner;
  const overheadSeconds = suiteRuns * org.shards * runner.shardOverheadSeconds;
  const computeUsd = ((testSeconds + overheadSeconds) / 60) * runner.costPerRunnerMinuteUsd;

  const retryRuns = org.runsPerMonth * org.firstAttemptFlakeRate * org.rerunsPerFlake;
  const retryComputeUsd =
    (retryRuns / runner.testsPerSecondPerRunner / 60) * runner.costPerRunnerMinuteUsd;

  const p50SuiteSeconds =
    org.testCasesPerSuite / org.shards / runner.testsPerSecondPerRunner +
    runner.shardOverheadSeconds;
  const p95SuiteSeconds = p50SuiteSeconds * p95ToP50Ratio;

  const developerWaitUsd =
    (p95SuiteSeconds / 60) * suiteRuns * org.blockedShareWhileWaiting * org.engineerUsdPerMinute;

  const flakeEvents = org.runsPerMonth * org.firstAttemptFlakeRate;
  const flakeTriageUsd =
    flakeEvents * org.investigatedFlakeShare * org.triageMinutes * org.engineerUsdPerMinute;

  return {
    computeUsd,
    retryComputeUsd,
    developerWaitUsd,
    flakeTriageUsd,
    totalUsd: computeUsd + retryComputeUsd + developerWaitUsd + flakeTriageUsd,
    p95SuiteSeconds: Math.round(p95SuiteSeconds),
    flakeEvents: Math.round(flakeEvents),
  };
}

Automating flake attribution in CI

A cost model is only actionable if you can attribute the failures. Emit Playwright's JSON reporter and reduce it into a quarantine manifest.

import { readFile, writeFile } from 'node:fs/promises';

interface PwResult { status: 'passed' | 'failed' | 'timedOut' | 'skipped'; duration: number }
interface PwTest { results: PwResult[] }
interface PwSpec { title: string; ok: boolean; tests: PwTest[] }
interface PwSuite { file?: string; specs: PwSpec[]; suites?: PwSuite[] }

function* walk(suite: PwSuite): Generator<PwSpec> {
  for (const spec of suite.specs) yield spec;
  for (const child of suite.suites ?? []) yield* walk(child);
}

export async function buildQuarantineManifest(
  reportPath: string,
  outPath: string,
  threshold = 2,
): Promise<void> {
  let report: { suites: PwSuite[] };
  try {
    report = JSON.parse(await readFile(reportPath, 'utf8')) as { suites: PwSuite[] };
  } catch (cause) {
    throw new Error(`Unreadable Playwright JSON report: ${reportPath}`, { cause });
  }

  const failures = new Map<string, number>();
  for (const suite of report.suites) {
    for (const spec of walk(suite)) {
      // A spec is flaky if it both passed and failed across retries in this run.
      const statuses = spec.tests.flatMap((t) => t.results.map((r) => r.status));
      if (statuses.includes('failed') && statuses.includes('passed')) {
        failures.set(spec.title, (failures.get(spec.title) ?? 0) + 1);
      }
    }
  }

  const quarantined = [...failures.entries()]
    .filter(([, count]) => count >= threshold)
    .map(([title]) => title)
    .sort();

  await writeFile(outPath, JSON.stringify({ generatedAt: new Date().toISOString(), quarantined }, null, 2));
  if (quarantined.length > 0) {
    console.warn(`[flake] ${quarantined.length} specs quarantined — open a tracking issue per entry.`);
  }
}

Sharding & Parallelism That Actually Saturates

Playwright CT — native sharding, correct load balance

Playwright shards by test count, not file count, so weights stay tight. Always pair --shard with fail-fast: false or one shard's failure cancels its siblings mid-run and corrupts your timing data.

// playwright-ct.config.ts
import { defineConfig, devices } from '@playwright/experimental-ct-react';

export default defineConfig({
  testDir: './src',
  testMatch: '**/*.ct.tsx',
  fullyParallel: true,
  retries: process.env.CI_NIGHTLY ? 2 : 0, // never retry in the PR lane
  workers: 4,
  reporter: process.env.CI ? [['json', { outputFile: 'ct-report.json' }], ['line']] : 'list',
  use: {
    trace: 'retain-on-failure',   // traces are 2-20 MB; retaining all of them is a storage bill
    video: 'off',                 // video is the single largest artifact line item
    screenshot: 'only-on-failure',
    ctPort: 3100,
  },
  projects: [{ name: 'chromium', use: { ...devices['Desktop Chrome'] } }],
});

Cypress CT — the parallelisation tax

Cypress's built-in --parallel load balancing requires a Cypress Cloud record key; without it you are partitioning spec files by hand, and component spec weights vary by 20× or more, so naive glob splitting produces stragglers. The open path most enterprises take is the cypress-split community plugin, which bins specs by historical duration.

// cypress.config.ts
import { defineConfig } from 'cypress';

export default defineConfig({
  component: {
    devServer: { framework: 'react', bundler: 'vite' }, // webpack cold start is ~1.7x slower
    specPattern: 'src/**/*.cy.tsx',
    supportFile: 'cypress/support/component.ts',
  },
  retries: { runMode: process.env.CI_NIGHTLY ? 2 : 0, openMode: 0 },
  video: false,
  screenshotOnRunFailure: true,
  e2e: { setupNodeEvents() {} },
});

Vitest browser mode — the cheapest lane

// vitest.config.ts
import { defineConfig } from 'vitest/config';
import react from '@vitejs/plugin-react';

export default defineConfig({
  plugins: [react()],
  test: {
    browser: {
      enabled: true,
      provider: 'playwright',
      headless: true,
      instances: [{ browser: 'chromium' }],
      // isolate: false reuses the module graph across files — big win in component
      // suites, but only safe once you have zero cross-test module state.
      isolate: true,
    },
    retry: process.env.CI_NIGHTLY ? 2 : 0,
    reporters: ['default', 'json'],
  },
});

# .github/workflows/ct.yml (excerpt)
strategy:
  fail-fast: false
  matrix:
    shard: [1, 2, 3, 4, 5, 6, 7, 8]
steps:
  - uses: actions/cache@v4
    with:
      path: node_modules/.vite
      key: vite-${{ hashFiles('**/pnpm-lock.yaml') }}
  - run: npx vitest run --shard=${{ matrix.shard }}/8

Decision Matrix

Choose Playwright CT if you already run Playwright for E2E and want one sharding model and one trace viewer. Choose Cypress CT if debuggability during component development is the primary constraint and cost is secondary. Choose Vitest browser mode if you are Vite-native and cycle time is the governing metric.

Operational Checklist

• retries: 0 in the PR lane; retries only in a separate nightly detection job.
• Emit machine-readable reports (JSON/JUnit) on every run, pass or fail — you cannot attribute flake you never recorded.
• Disable video; use trace: 'retain-on-failure'; set artifact retention to 3 days.
• Cache the Vite dep-optimisation directory (node_modules/.vite) keyed on the lockfile hash.
• Pin browser versions across the whole runner fleet; drift is a silent 5% flake source.
• Alert on p95 suite wall time, not p50 — p50 hides the bad days.
• Never set workers above vCPU count.
• Quarantine specs after 2 flake events in 7 days, with an owner and a deadline.

Pitfalls to Avoid

1. Benchmarking on a laptop. Thermal throttling and a warm dep cache produce numbers 40% optimistic versus a cold CI runner.
2. Reporting p50 only. The whole point of a p95 target is that the median run is already fine.
3. Treating retries as a stability strategy. They hide the signal you need to fix the 22% of flakes caused by shared module state.
4. Setting `isolate: false` before your tests are hermetic. You will trade 30% throughput for a 4× flake rate.
5. Assuming Cypress `--parallel` is free. It requires a Cloud record key; budget for it explicitly or adopt a split plugin.
6. Ignoring triage cost in the business case. A migration that saves $50/month in compute but adds 0.3% flake is a $20,000/month regression.

Sources & Technical References

• Playwright — Component Testing: https://playwright.dev/docs/test-components
• Playwright — Sharding: https://playwright.dev/docs/test-sharding
• Playwright — Parallelism and workers: https://playwright.dev/docs/test-parallel
• Playwright — Trace Viewer: https://playwright.dev/docs/trace-viewer
• Cypress — Component Testing overview: https://docs.cypress.io/guides/component-testing/overview
• Cypress — Parallelization (requires record key): https://docs.cypress.io/guides/guides/parallelization
• Cypress — Retry-ability and command queue semantics: https://docs.cypress.io/guides/core-concepts/retry-ability
• Cypress — Configuration reference (video, retries, screenshotOnRunFailure): https://docs.cypress.io/app/references/configuration
• Vitest — Browser Mode guide: https://vitest.dev/guide/browser/
• Vitest — Browser configuration reference: https://vitest.dev/guide/browser/config
• Vitest — CLI (--shard, --sequence.shuffle): https://vitest.dev/guide/cli
• Vitest — pool and isolation: https://vitest.dev/config/#pool
• Vite — Dependency pre-bundling and cache: https://vitejs.dev/guide/dep-pre-bundling
• MSW — Node integration and request lifecycle: https://mswjs.io/docs/integrations/node
• GitHub Docs — Actions billing and per-minute rates: https://docs.github.com/en/billing/managing-billing-for-your-products/managing-billing-for-github-actions/about-billing-for-github-actions
• GitHub Docs — Larger runners and pricing: https://docs.github.com/en/actions/using-github-hosted-runners/using-larger-runners
• Actions Runner Controller (self-hosted runner economics): https://github.com/actions/actions-runner-controller
• Dean, J. & Barroso, L.A. — The Tail at Scale, Communications of the ACM, 2013: https://research.google/pubs/pub40801/
• Google Testing Blog — Flaky Tests at Google and How We Mitigate Them: https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html
• MDN — PerformanceObserver and long-task measurement: https://developer.mozilla.org/en-US/docs/Web/API/PerformanceObserver
• Testing Library — Async utilities and waitFor semantics: https://testing-library.com/docs/dom-testing-library/api-async