Reporting for Stakeholders

The team has test results. What the team needs is one URL that answers, without a single click into a log: is quality improving or decaying? That is the dashboard's whole job — and it is built from five numbers computed the same way on every run.

Publish the report as an artifact of the pipeline, not a folder on someone's disk:

npx playwright test --reporter=json --output-dir=test-results
npx allure generate allure-results --clean -o allure-report
# serve locally while iterating
npx allure open          # http://localhost:5050

In CI, upload allure-report/ as an artifact and mirror it to a gh-pages branch nightly so the stable link — https://acme.github.io/capstone-automation/ — always shows the latest full run. Allure gives you history for free once allure-results are archived between runs, which is the requirement that separates a trend dashboard from a snapshot. Tooling choice is smaller than it looks: Allure plus static hosting is the honest default for a capstone (one container or one branch, no database), while Grafana with a Prometheus data source earns its setup when quality metrics must sit next to service SLOs on one board, and TestOps-style servers add collaboration features at the price of something else to operate.

The five numbers, and the definitions that stop arguments about them:

Every definition has a way to be gamed, so fix it in the formula. Skip 200 obsolete tests and a raw pass rate jumps while coverage silently fell — exclude skipped. Retry a flaky test twice and the pass rate never moves — count flaky separately and put the percentage on the wall. The one thing that matters: a metric whose formula is not written down will be quoted with three different meanings in the same meeting.

Per-test detail is where the dashboard earns trust with engineers. Allure's Flaky tab lists the offenders with their recent history — checkout-guest-payment failed 7 of its last 30 runs and passed on retry each time. Attach the owner (from a test.info().annotations entry your fixture adds) and the triage command, and the flaky list becomes a work queue instead of a complaint. Annotate trend lines with release markers: the flaky spike at run 4,112 lines up exactly with the merge that introduced parallel worker state in the fixtures, and a graph makes that visible in two seconds while a wall of results never would.

Keep the dashboard honest by making it fully automated: generated in the pipeline, published on schedule, containing no manually pasted numbers. The moment someone edits a figure before a review, the URL stops being trusted and email attachments come back. The teams that get this right stop arguing about whether quality is fine and start arguing about which panel to fix first — a much better argument to be having. Your dashboard is for the team's daily reading; the next piece splits the same underlying data into what each stakeholder group actually needs.

Every quality report you have ever read badly was the same report sent to everyone: wall-of-text test names for a VP, "coverage went up" for an engineer who needed the failing diff. The capstone's reporting is one dataset rendered three ways, because the audience's question defines the report, not the data you happen to have.

The engineer view is generated by the pipeline, not written by a human. A failing smoke test posts the test title, the shard, a link to the trace artifact, and a copy-paste reproduction:

npx playwright test checkout --grep "guest pays with card" --headed
npx playwright show-trace test-results/checkout/trace.zip

The manager view is the dashboard from Wiring the Capstone into CI plus a weekly digest that reports movement, not levels: flaky 1.4% → 0.8% after the fixture rewrite; PR feedback p95 at 12:40 against the 15-minute budget; 3 tests in quarantine, oldest ticket QAI-1183 at 9 days. Leadership gets the trend in the language of risk: escape rate 8%, down from 15% two quarters ago; the two production incidents this quarter both mapped to the payment-redirect journey, where a new contract test now runs; no open release blockers.

Two rules keep the three renderings consistent. First, one source of truth: every view derives from the same pipeline artifacts and metric definitions, so no audience can be shown 98% while another is shown 96%. When two numbers in one meeting disagree, trust in all reporting dies at once. Second, the weekly and release-readiness digests are generated, never compiled by hand:

- name: Release readiness
  if: github.event_name == 'schedule'
  run: node scripts/report/release-readiness.js   # posts to Slack webhook
  env:
    SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
    BASELINE_ESCAPE_RATE: 0.15   # quarter's starting point

The script reads the JUnit XML from the nightly run, joins escape-rate history from a small JSON store, and posts a fixed template — pass rate, flaky rate, duration, top 3 offenders with owners, quarantine count, escape-rate delta. It runs in about 40 seconds, and nobody ever assembles the update manually, which is the only reason it keeps being sent in week 30.

The template is short on purpose — five lines plus links, because a digest that needs a meeting to parse stops being read. Every number carries its comparison (flaky 0.8%, ↓ 0.6 pts) and every red item carries an owner and a due date. When a metric crosses a threshold, the script appends the relevant dashboard panel link, so the digest points at the evidence rather than replacing it — the digest is the index, the dashboard is the record, and the trace is the truth.

The skill interviewers are testing with "how do you communicate quality to non-technical stakeholders?" is translation with discipline: lead with risk and trend, in bullets, with a date and an owner on anything actionable — and keep the trace links available for the one engineer in the room who asks. The third audience-facing move is the one nobody plans for: when something does escape, the report has to explain why, which is the root-cause work in the next chapter.

A production incident is filed: customers report their coupon applied twice and orders were discounted 40%. The framework had 1,840 passing tests. The question that determines whether this repeats is not "who missed it" but which layer of the defense was absent — and the answer is recorded, not debated.

Start with a metric that can be argued about precisely. Define escape rate once and put the formula in the dashboard:

escape rate = defects found in production / (defects in production + defects caught pre-release)

this quarter:  4 / (4  + 46)  = 8%
last quarter:  9 / (9  + 51)  = 15%

An 8% that fell from 15% tells a story the raw bug count never could. Pair it with a severity split — a P1 escape and a cosmetic escape are not the same defense failure — and with "escapes per release" so a quarter with two hotfixes cannot hide behind a big release batch.

Then classify. Every escape gets a root-cause category at a 30-minute blameless review, and each category has a prescribed suite action:

Walk the double-coupon case through the table. Five whys: the coupon doubled → the API allowed stacking → POST /coupons/apply had no idempotency or uniqueness constraint test → the contract suite covered status codes and schemas, not business rules → business rules lived only in UI tests, where the tester assumed the API checked them. The class is no test case at the API layer, and the action is concrete:

test('applying the same coupon twice returns 409 @contract', async ({ api, customer }) => {
  await api.applyCoupon(customer, 'SAVE40');
  const res = await api.applyCoupon(customer, 'SAVE40');
  expect(res.status()).toBe(409);
  expect((await api.getOrder(customer)).discount).toBe(40);   // not 80
});

That test is committed with the incident ID in its body, and the process fix — business-rule cases must exist at the API layer before a journey is called covered — goes into the review's second output. One test plus one process change per escape is the ritual; either alone is the same incident wearing a calendar costume.

The release-readiness report closes the loop, because prevention has to arrive before the release, not after it:

node scripts/report/release-readiness.js --tag v4.2.0
# → pass rate 98.6% · flaky 0.7% · quarantine 3 · blockers 0
# → escape trend 15% → 11% → 8% · readiness: GO with notes

Automated from the nightly artifacts, posted to the release channel the morning of a deploy decision — the same pipeline data as the dashboards and digests, rendered for the one meeting where quality is decided. When an escape still happens, the review has made the response mechanical instead of political: classify, add the test, change the rule, update the trend. That is also the raw material for the final chapter, where the framework's own future gets reviewed and demonstrated in Maintenance, Review, and Demo.