Debugging Failing Tests

A nightly run reports: checkout › applies promo code — timeout after 30s. Your first instinct is to re-run it. Resist. A failure that cannot be reproduced on demand is a specimen, and the specimen's evidence is perishable — reruns, restarts, and cleaned-up CI workspaces destroy it. The disciplined order is: open the artifacts, reconstruct what happened, form a hypothesis, then attempt reproduction.

The trace is the primary artifact. With trace: 'on-first-retry' (or --trace on), Playwright records, for every action: a before/after DOM snapshot, console messages, network requests, and screenshots. npx playwright show trace trace.zip opens a timeline you scrub like a video:

action            snapshot shows                side channels
─────────────────────────────────────────────────────────────
goto /checkout    page as loaded                200s except /api/promos → 404
click Apply       spinner visible               console: "Failed to load resource: 404"
waitFor visible   still spinner at 30s          POST /api/promos payload visible

Read it backwards from the failing assertion: what did the page look like at the last snapshot, and what do the side channels say? In the example above the 404 on /api/promos appears one action before the timeout — the failure is not a wait problem at all, it is a backend regression, and the trace proves it in thirty seconds. Notice what this replaced: a twenty-minute session of re-running and guessing.

Screenshots and video complement, not replace, the trace. A screenshot (screenshot: 'only-on-failure') shows one frozen frame — great for layout and visibility failures, useless for ordering questions (did the spinner come before or after?). Video shows motion without DOM internals — useful for gesture and animation bugs, expensive to review, and a poor substitute for the snapshot timeline. The reading order that works: failure message → action at the boundary → snapshot at that action → network/console for the why.

In Selenium, assemble the same evidence manually. There is no built-in trace, so configure artifacts up front — failure screenshots in the framework's after-each hook, browser console logs via LoggingPreferences/goog:loggingPrefs, and the network log via Chrome DevTools Protocol Performance domain or a HAR export. Many teams wrap this in one helper: captureArtifacts(driver, testName) writing screenshot.png, console.log, and network.har into a per-test directory keyed by the test name, attached to the report (Allure, JUnit XML) — the goal is that every red run produces the same evidence bundle as a Playwright trace, without the engineer remembering to take one.

The diverging-step habit. Every failure is a moment where the observed world stopped matching the expected one; the trace's job is to find that moment precisely. Not "checkout is broken" but "promo lookup returned 404 from commit 7f3a2b1". This precision is also what makes the next phase — classification — possible, because a 404 from a new commit and a strict mode violation in your own locator are different kinds of events with different owners. Keep one more habit in mind while reading traces: timestamps. If the trace shows a 2.3-second gap between click and render that is absent locally, you have already half-diagnosed an environment or load issue before touching any code.

"The test failed" is not a diagnosis. Before anyone fixes anything, the failure needs a class, because each class has a different owner, a different workflow, and a different lesson. The triage decision tree runs on evidence from the trace, in this order:

Step 1 — did the application behave per its specification? Open the trace at the failing step and compare observed behavior to intended behavior ignoring the test entirely. Three outcomes:

App did the wrong thing            → PRODUCT DEFECT
  e.g. promo field rejects valid code, UI shows wrong total, 404 from a new commit
App did the right thing, test expected wrong → TEST BUG
  e.g. assertion targets stale text, locator grabbed wrong element, timing assumption
App's intent unknown/never reached → go to Step 2

This ordering is the whole discipline: judge the app first, on its own terms. Teams that triage in the other direction explain away real regressions as "flaky tests" — the most expensive category error in QA, because the suite has now detected a defect and then been told to ignore it. A product defect gets a proper bug report (steps, expected vs actual, trace/screenshot attached, commit range from the run history) and a test-side guard if the defect escaped review.

Step 2 — does it depend on environment? If the app's behavior is indistinguishable from expected, ask what differs: browser version on the failing node, base URL and feature flags, data state, network conditions, load. Cheap discriminators: does it fail every run on one node but never others (environment), does the trace show a 5-second network gap absent locally (load/network), did a flag change flip behavior (config). Environment issues route to the platform owner with evidence, not to the test author — and if the environment is genuinely flaky, that is the quarantine conversation from Waits and Flakiness Control, not a retry conversation.

Step 3 — is it order-dependent? Passes alone, fails in the suite: leaked state, shared data, an inter-test dependency. The fix is isolation (per-test data, independent setup), and the diagnostic tool is running the test alone versus after the file — a two-minute experiment that reclassifies a large fraction of "random" failures.

A compact tree, worth pinning in the triage doc:

trace shows app ≠ spec? ── yes ──▶ PRODUCT BUG  → defect report + regression test
        │ no
        ▼
differs by node/config/load? ── yes ──▶ ENVIRONMENT → platform ticket + evidence
        │ no
        ▼
passes alone, fails in suite? ── yes ──▶ ORDER-DEPENDENT → isolate data/setup
        │ no
        ▼
TEST BUG → fix test code, hold it to production standards

Two reporting habits complete the loop. First, every classification gets recorded (a label in the issue tracker: product, test, environment, order) — the monthly distribution is your quality dashboard: a rising test share means the suite is rotting; a rising product share near release means risk. Second, the test that found a real product bug never gets deleted after the fix — it becomes the regression test, which is the direct justification for its existence in the next sprint's planning.

"Reproduces locally" is the goal of the debugging session, and "cannot reproduce" is where bad fixes are born. Local reproduction is not luck; it is matching conditions until the failure reappears on demand. The checklist that makes it mechanical:

1. Browser/version   same Chromium major as CI (npx playwright install)
2. Config            CI env vars set identically (BASE_URL, flags, headless)
3. Data              same starting state — reseed, use the same fixture version
4. Load              run the suite's parallelism locally (--workers 8) if contention suspected
5. Seeds/timing      repeat-each=10 to convert 'once in 20' into a repeatable specimen

Headed serial runs (--headed --workers=1) answer "what does the flow do"; the parallel repeat answers "what makes it break". When a failure genuinely will not leave CI, the honest moves are: keep the trace as the specimen, instrument the test to fail louder (assert with descriptive messages, capture an extra screenshot at the suspicious step), and add a temporary debug artifact to the next CI run — not to re-run blind. In Selenium the same checklist applies with manual artifacts: run the same driver/browser version locally (Docker images pinned to CI's browser are the reliable trick), and set the same env-driven base URL.

The other half of debugging is the fix, and test code deserves production discipline. The rationale is economic: a failing test blocks every engineer's merge; a bad test fix breaks silently later. Concretely:

Small, single-purpose commits      one flake fixed per PR, reviewable diff
Tests reviewed like app code       reviewer checks the FIX did not weaken the assertion
No assertion deletion to go green  removing a check is a decision, not a fix
Red → green proof                  PR description shows the failing run, then the passing one

The cardinal sin deserves its own line: fixing a test by making its assertion vacuous — exists instead of text equals, dropping the toBeVisible, adding first() to swallow strict-mode ambiguity — converts a working detector into decoration. A reviewer's litmus question: "would this change have caught the original bug?" If not, it is not a fix; it is a surrender with a green checkmark.

Leave the code better than the failure found it. Each debugging session should pay a small dividend: a clearer assertion message (so the next failure explains itself), one brittle locator promoted to the convention's semantic form, one missing wait replaced with a condition, one helper that captures evidence automatically. This is how suites age well instead of accumulating scar tissue — a hundred such dividends is the difference between a suite people trust and a suite people re-run until green. Budget for it explicitly if needed: some teams reserve the last twenty minutes of a debugging session for exactly this cleanup, because the pressure to move on is real and the interest on skipped cleanup compounds. It is also the natural transition to the structural work of the next phase. When locators and waits are handled at the right layer, debugging sessions shrink, because each screen's knowledge lives in one maintained place: Page Object Basics.