Maintenance, Review, and Demo

A suite that nobody maintains does not fail suddenly — it decays. Selectors break one by one, fixtures outlive the data they seed, and eighteen months later the team reruns tests until green and trusts none of them. The countermeasure is not effort, it is a calendar: recurring maintenance items with owners, cadence, and visible evidence.

Run the loop on two clocks. Weekly, a 30-minute triage: every red build from the last seven days is classified as product bug, test bug, flaky, or infrastructure, and each class gets an owner before the meeting ends. The service level is simple enough to remember — every red build has a named owner within one business day — because an unowned red build teaches the team that red means nothing.

The taxonomy needs evidence or it becomes storytelling. Classify from what the pipeline already stored: the failing assertion in test-results/, the trace for whether the DOM or the API drifted, the nightly quarantine report for repeats. Signatures are worth writing down in the team wiki: TimeoutError: locator.resolve on a rock-solid test-id usually means the page never rendered because a dependency call failed — an API problem wearing a UI costume — while a sudden 403 on an API spec that passed yesterday points at the account fixture or an expired token, not the endpoint. Ten minutes with a trace beats an hour of theorizing, and the classifications you collect here feed the escaped-defect reviews later.

Monthly, a dependency and debt pass driven by automation rather than heroics:

// .github/renovate.json
{
  "schedule": ["before 6am on monday"],
  "packageRules": [
    { "matchPackagePatterns": ["playwright"], "groupName": "playwright stack",
      "automerge": false },
    { "matchUpdateTypes": ["minor", "patch"], "automerge": true,
      "automergeType": "pr", "requiredStatusChecks": ["capstone-smoke"] }
  ]
}

Minor and patch bumps automerge only after the smoke lane proves them — the gate you built now protects the framework itself. Majors (Playwright 1.4x → 1.5y) get a manual PR, a read of the changelog for locator and config deprecations, and a full-suite run before merge.

Each volatile asset in the framework has a maintenance item with the right frequency:

Selector churn gets special attention because it is the decay vector you can measure. The page objects from Designing the Capstone Framework localize it; you go further and track its size:

grep -rn "getByTestId\|getByRole" src/ui | wc -l      # the surface area
grep -rln "css=.*div\." src/ui                        # the stragglers to purge

When the second number is non-zero, a refactor is scheduled — brittle chains are debt with interest, and the interest is paid in triage hours.

Budget the whole loop honestly: reserve about 10% of sprint capacity for framework maintenance, and defend it in the same review where you defend features. The argument is arithmetic, not sentiment: a 38-minute suite that rots gains retry time and loses trust, and rebuilding trust costs far more than the 10% ever did. A routine proves itself by what it prevents — and by how cleanly it lets you remove things, which is the discipline the next chapter covers.

There is a test named legacy-cart-2019.spec.ts that has failed on 60% of runs since March. Someone reruns the job until it passes. Nobody remembers what it protects. This is how suites die — not by deletion, but by learned helplessness around tests nobody is allowed to touch. The alternative is deliberate retirement: every removal is a decision with criteria, an owner, and a written record.

Decide with an explicit policy, applied at the monthly pass:

Quarantine is a state, not a .skip someone forgot. In Playwright it is a tag plus two pipelines:

# the gate — quarantined tests are excluded from the merge verdict
npx playwright test --grep-invert @quarantine

# the nightly — quarantined tests still run, and their failures are reported
npx playwright test --grep @quarantine --reporter=github

The quarantine tag requires a ticket in the test's annotation (test.info().annotations.push({ type: 'ticket', description: 'QAI-1183' })), and the nightly quarantine report appears in the weekly digest with the oldest ticket's age. A quarantined test with no ticket and no age is just deleted already — the ticket is what stops quarantine from becoming a graveyard.

Track the quarantine count itself on that weekly digest; it is the honest version of the pass rate. A quarantine that grows every month is a suite being pruned of its hard problems, while one that empties is a team paying its debts down. Put a ceiling on the state — no more than 2% of the suite quarantined at once — so it stays temporary by construction instead of by good intentions, and make the ceiling a gate you actually look at in the retrospective.

Retirement itself is a commit that touches three things: the test, the changelog, and nothing else — no sweeping refactors in the same PR, so the removal stays reviewable:

## TEST-CHANGELOG

### 2026-09-14 — retired `legacy-cart-2019.spec.ts` (14 tests)
Reason:   cart-v1 removed in release v4.2 (flag CART_V2=100%)
Replace:  none — journey superseded by `cart-v2.spec.ts` (API-seeded)
Coverage: 1,842 → 1,828 declared tests; spec matrix row "Cart v1" removed
Ref:      commit 9f3c1ab · QAI-1402

That file is the answer to the auditor's inevitable question — "why did declared coverage drop last month?" — answered in 30 seconds without archaeology. It is also the answer interviewers probe when they ask how you handle obsolete tests: delete only with a record, keep the strategy matrix and the tests in the same commit, and never let a coverage number move without a written cause.

The trust argument is the deeper one. Teams stop believing coverage percentages the first time a number drops silently; they stop believing the suite the first time a critical test is deleted "because it was flaky". Deliberate retirement — criteria applied uniformly, a quarantine that expires into a decision, a changelog auditors can follow — is what makes the remaining 1,828 tests mean something. Maintenance keeps the suite alive; retirement keeps it true. What is left is to prove both to the people who fund them, which is the retrospective and demo in the final chapter of the roadmap.

The framework is built, wired, reported, and maintained. Two obligations remain, and they are the ones that decide whether the work is remembered: an honest retrospective on the framework itself, and a demo that proves its value to people who will never read its code.

Feed the retrospective with data, not impressions. Pull the last quarter's numbers straight from the pipeline artifacts behind the dashboards — the shared dataset from Reporting for Stakeholders:

suite duration          38m → 12m30s wall-clock (4 shards)
flaky rate              2.1% → 0.7%
PR feedback p95         31m → 12m40s
escaped defects/qtr     9 → 4 (escape rate 15% → 8%)
maintenance hours/wk    6.0 (selector churn 41%, env issues 22%)

Then run a 60-minute session with the three questions that produce output instead of catharsis: what is working that we must not break, where does the framework actively cost us, and what is the next phase? Each gap becomes a ranked proposal with an estimate — "replace 140 UI-clicked setup steps with API fixtures: −6 minutes suite duration, ~2 days of work" beats "reduce flakiness" as a backlog item because it can be accepted or rejected on the spot. The output is three to five committed proposals with owners and dates; a retrospective whose actions live in someone's notebook did not happen.

The demo is an engineering exercise with one success criterion: the audience sees a red test produce evidence and a green test prove a journey, live. Structure it in five beats, about eight minutes total:

# 1 — repo + README, then first green, from a clean clone (pre-warmed)
npm ci && npm run test:smoke          # 12 passed (6.4s)

# 2 — the full run against the demo tag, 4 shards in the CI UI
npx playwright test --shard=1/4

# 3 — the deliberate failure: flip the seeded bug, rerun one test
npx playwright test checkout --grep "applies coupon once"
#   → expect(409).received(200) + trace.zip in the artifact

# 4 — evidence on screen
npx playwright show-trace test-results/checkout/trace.zip

# 5 — the reports: dashboard trend line, then the Slack readiness post

The seeded bug is the keystone: a known defect (your double-coupon regression from the escaped-defect process makes a perfect subject) that the suite catches on camera. Nothing else you say about reliability lands like a red build turning into a named assertion, a stack, and a downloadable trace in 40 seconds.

Live demos fail in predictable ways, so engineer them: run against a pinned demo tag, never main, so a teammate's merge cannot sabotage you; pre-warm node_modules and browsers on the demo machine; and have the pipeline's video artifact queued as the fallback. State the reliability numbers up front — 0.7% flaky over 30 days — so the audience knows what they are watching is typical, not lucky.

Prepare for the two questions that always follow. "What happens when the framework itself breaks?" — answer with the maintenance loop: Renovate PRs gated by the smoke lane, weekly triage with a one-business-day owner SLA, quarantine ceiling at 2%. "What would you do next?" — answer with the retrospective's ranked proposals, by name and estimate, because they were committed before the demo, not invented during it.

Expect the question that ties the whole capstone together: "walk me through your framework." Answer it in the roadmap's own order — three layers and their boundary rule, one config entry point, two CI triggers and the gates they enforce, one dataset rendered for three audiences, a maintenance loop with a retirement policy — then hand over one metric, such as PR feedback p95 dropping from 31 minutes to 12:40. Layers, gates, evidence, and upkeep, in one breath: that is the difference between a folder of tests and a framework.