Parallelization and Flaky Test Management

The suite takes 40 minutes and nobody runs it before merging, which is the same as not having it. Parallelization is the fix, and it is a scheduling problem more than a testing one — which matters doubly here, because the gate from Quality Gates and Merge Policy is only as trustworthy as the suite behind it. Split the work so each worker gets a roughly equal share, keep workers isolated, and make sure the slowest shard — not the average — sets your feedback time.

Most runners shard directly. Playwright takes a fraction, and the matrix fans it out:

npx playwright test --shard=1/4    # worker 1 of 4; deterministic split of the file list

# .github/workflows/e2e.yml
jobs:
  e2e:
    strategy:
      fail-fast: false
      matrix: { shard: [1, 2, 3, 4] }
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20, cache: npm }
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test --shard=${{ matrix.shard }}/4 --reporter=blob
      - uses: actions/upload-artifact@v4
        with: { name: blob-${{ matrix.shard }}, path: blob-report/ }
  report:
    needs: e2e                    # merge the shards into ONE report
    if: always()
    steps:
      - run: npx playwright merge-reports --reporter html ./blob-report

The blob reporter plus a merge job is not decoration: without it, four shards produce four HTML reports and nobody knows the total pass rate. One suite, one report, however many workers produced it — a principle you will want again in Test Reporting and Dashboards.

Balancing is where the minutes are won or lost. Fixed splitting (files[i % 4]) on a file-ordered manifest gives shard 1 every slow end-to-end spec and shard 4 a quiet ride — your 40-minute suite becomes 4 × 18 minutes instead of 4 × 10. Two cures: sort by historical duration descending, then interleave round-robin, or run shards from a single queue where each worker pulls the next file (dynamic scheduling self-balances at the cost of a coordinator). Measure it: publish duration per shard, and treat max shard duration as the number you optimize — averages hide the shard everyone waits for.

Containers make the isolation explicit and the scaling elastic: docker compose up --scale worker=4 with each worker running the identical image against its own database, or a CI matrix where every job is already a fresh VM. The two costs to budget are startup (a browser install per worker — cached, per the caching rules from Integrating Tests into CI) and shared infrastructure capacity: your staging database, test API, and third-party rate limits do not scale with your worker count. Four workers against an unscalable backend turns parallelism into a queue of 503s — which is a concurrency bug in the environment, and it looks exactly like a flaky test until someone graphs failures against worker count.

Tune worker count against the resource that saturates first, and it is usually not the CPU. Browser suites want roughly one context per two cores and two GB of RAM; past that, pages render slower and condition waits lengthen, which converts a parallelism win into a flakiness report. Start at maxWorkers = min(cpus / 2, 4), mark genuinely serial tests (visual comparisons, anything touching a global fixture) to run alone, and let the framework split contexts where it can instead of processes. On a paid runner the arithmetic is identical but the bill is not: every worker is billed for the whole job, so the sweet spot is the smallest worker count that still lands inside the feedback budget.

Parallelism does not create flakiness — it reveals it, by hitting timing windows and shared resources far more often per hour. Once you know a test sometimes fails, you face the policy decision every team eventually argues about: do you retry? The answer is yes, exactly once, and you record the outcome — because a retry that erases the evidence is indistinguishable from a test that passed.

// playwright.config.ts
export default defineConfig({
  retries: process.env.CI ? 1 : 0,   // one retry in CI, none locally — local must stay honest
  forbidOnly: !!process.env.CI,       // no focused tests left behind
  reporter: [["html"], ["json", { outputFile: "results.json" }]],
});

# mocha / jest equivalents
npx mocha --retries 1 --reporter mocha-multi-reporters
npx jest --maxWorkers=4                 # jest has no built-in retry: rerun failures once in a second pass

A retry changes the meaning of a run, so the report must distinguish three states — passed, failed, passed-after-retry (flaky) — and count the third separately. Playwright does this natively (its HTML report marks a test flaky), and any framework without it needs a small post-processing step over the JSON results: flaky = tests where attempt > 1 && finalStatus === 'passed'. Without that distinction you have two untrustworthy readings: a suite where "0 failed" hides ten tests that needed a second chance, or a suite where every flake is reported as a failure and the team learns to ignore red.

Retries as policy need three boundaries:

• One attempt, not three. Retrying until green converts a 5% flake into a 0.01% visible failure while multiplying suite time — the defect still ships.
• Retry only on CI. Locally a developer should see the failure first time; otherwise they debug against a run their machine will never reproduce.
• Route persistent flakes to quarantine, not to more retries. A test that retries more than a couple of times per week is a broken test: the follow-up is the quarantine list, with an owner and a date.

Flake rate itself is a measurable quantity — flaky tests / total tests over a window — and it belongs on the same dashboard as pass rate. A rising curve after a parallelization change is diagnostic: it usually points at the shared resources in the third chapter rather than at newly fragile assertions. Track it per suite and per worker count, and you will see the exact worker threshold where contention starts. Before quarantining anything, collect evidence rather than opinions: run the suspect test repeatedly — npx playwright test checkout --repeat-each=50 — and read the distribution. Failures at ~15% with a different error each time usually mean something is being waited for instead of asserted; failures at 100% after a data-seed change are a real regression mis-filed as flake; failures clustered at the start of a run point at cold caches or missing fixtures; failures that appear only on the slowest worker point at a hidden timeout. The three reflexes people reach for first — a larger timeout, an extra sleep, another retry — treat the symptom while making every future failure slower to find.

Retries keep the pipeline usable while that gets fixed; they are a seatbelt, never the steering.

Every parallel-execution bug is the same bug: two workers using one thing. The failure appears random because it depends on timing, which is exactly why parallelization is such an effective flakiness detector — it converts hidden dependencies into daily failures. Hunt them in four places.

// ❌ fixed port — worker 2 dies with EADDRINUSE the moment workers overlap
const server = app.listen(3000);
// ✅ let the OS assign one — then read it back
const server = app.listen(0);
const port = server.address().port;

// ❌ shared temp path — results overwrite each other, order decides the winner
writeFileSync("/tmp/report.json", payload);
// ✅ per-worker directory
const dir = fs.mkdtempSync(path.join(os.tmpdir(), `worker-${workerId}-`));

// ❌ shared rows — one worker deletes "seed data" another is asserting against
await db.query("DELETE FROM orders");
// ✅ per-worker namespace
await db.query(`CREATE SCHEMA IF NOT EXISTS run_${workerId}`);

The database deserves its own paragraph because it is the most common culprit: fixed seed data, TRUNCATE in a global beforeEach, a shared test account whose cart one worker empties while another checks it, and auto-increment ids that tests assume start at 1. The fixes are structural — one schema/database per worker, unique entities per test (order-${Date.now()}-${workerId}), and cleanup scoped to what the test created — not a sleep between workers. Next come time and randomness (tests pinned to "today" or to a mocked clock without restoring it, random seeds not captured for replay), then file paths, caches, and singletons like a module-level browser or API client shared across workers.

Once parallel runs are green most of the time, the leftovers need an owner rather than a shrug. A quarantine is a list, not a skip:

# quarantine.yaml — reviewed like code, visible in the report
- test: "checkout/apply-promo.spec.ts :: SAVE10 halves the total"
  since: 2026-09-28
  owner: "@sam"
  reason: "fails ~15% on api-2 after coupon seed migration; suspected race"
  expires: 2026-11-15          # after this date the test runs again and fails the build
  ticket: "QA-772"

A quarantined test keeps running (in a non-blocking lane) so you still learn whether it got worse, but its failures don't block merges — and the expires field is what stops quarantine from becoming a graveyard: at expiry the test rejoins the required suite, which creates the pressure to actually fix it. Never delete a flaky test without a recorded decision; deletion is how suites shrink until only the easy assertions remain. Two supporting habits keep the list short. Generate data inside the test — an order created, asserted, and torn down by the same worker cannot collide with anything, which removes an entire class of parallel failures at design time rather than in triage. And make the policy self-enforcing: a CI step that fails when quarantine.yaml holds an expired entry, plus a widget showing count and age, so "we should fix those someday" becomes a dated obligation that appears in the same report everyone already reads. Both convert vague debt into something with a name and a date on it.

Report quarantine health beside pass rate: count, age, and expired entries — a list that grows every sprint is telling you that either the tests or the code beneath them are structurally unreliable. The dashboard that keeps all of this visible is Test Reporting and Dashboards.