Test Reporting and Dashboards

CI already produces results — JUnit XML, a wall of green and red — and almost nobody reads them, because a raw XML file is not an answer to the question a person has: what failed, why, and is it my change? An Allure report turns those results into a browsable HTML page with suites, histories, retries, attachments, and trends, and publishing it is a handful of commands.

npm i -D allure-js-commons allure-commandline allure-playwright

// playwright.config.ts — the reporter writes raw results, CI turns them into HTML
export default defineConfig({
  reporter: [
    ["list"],
    ["allure-playwright", { detail: true, outputFolder: "allure-results" }],
    ["junit", { outputFile: "results/junit.xml" }],   // keep the XML for machines
  ],
});

npx allure generate allure-results -o allure-report --clean
npx allure open allure-report            # local preview
npx allure serve allure-results          # one-shot: generate + open + expire

The distinction between the two outputs is worth internalizing: allure-results is the machine format (one JSON file per test plus attachments), and allure-report is the generated site. CI consumes the former to produce the latter, then publishes it. Two wiring details decide whether anyone ever sees it:

- name: Build and publish the report
  if: always()                    # reports matter most when the run failed
  run: npx allure generate allure-results -o allure-report --clean
- name: Deploy report
  if: always()
  uses: peaceiris/actions-gh-pages@v3
  with:
    github_token: ${{ secrets.GITHUB_TOKEN }}
    publish_dir: ./allure-report
- uses: actions/upload-artifact@v4
  with: { name: allure-results, path: allure-results/ }

Publish it at a stable URL (https://qa.example.com/allure/) served by CI, not as a downloadable ZIP — an artifact someone must fetch and unzip locally gets opened once. if: always() is the other non-negotiable: a report generated only on success documents the happy runs and omits exactly the failures people need. Keep junit.xml alongside for machines (gates, dashboards, analytics parse XML/JSON, not HTML), and cache or copy allure-results/history between runs — that folder is what buys you the trend charts in the next chapter. Playwright's own html reporter is a perfectly good lighter alternative for a single-project suite; Allure earns its keep with attachments, history, categories, and cross-framework aggregation. Stamp the run with metadata, or trends and drill-downs have nothing to join on. Allure reads two small files dropped into allure-results:

cat > allure-results/executor.properties <<EOF
name=github-actions
buildName=${GITHUB_SHA::8}
buildUrl=${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}
reportName=Shop API and E2E
EOF
echo "base_url=$BASE_URL" > allure-results/environment.properties
echo "browser=chromium"  >> allure-results/environment.properties

Those lines make a report clickable back to the exact CI run that produced it — one click from a failed assertion to the logs, the diff, and the commit — and they record which environment the results came from (base URL, browser, OS), so nobody relitigates whether a failure was on staging or on production. It is a handful of lines, and it is the highest-value part of the whole configuration.

Whatever the tool, the habit is the same — every run produces a report, the report survives the run, and its URL never changes. Teams that adopt this once stop debating whether a run was "clean": the answer is one link away, with the same address today as it had six months ago.

One run is a snapshot; ten runs are a story. The trend widgets — pass rate, duration, flaky count — are the part of a report that leadership reads and the part that silently rots if the underlying identity of a test is unstable. Getting them right requires understanding what the history is keyed on.

Allure builds trends from historyId, derived from the test's name and parameters (other tools use suite + title, or a stable id you supply). The practical consequence: rename a test and its trend resets — the chart shows a new test appearing with one data point while the old one stops, and six months later nobody can answer "is checkout getting flakier?". The discipline that prevents it: name tests by behavior (SAVE10 halves the total, not test 17), keep renames rare and deliberate, and where a framework allows it pass an explicit stable id. Parameterized tests need stable parameter values too — an id embedded in the parameters fragmenting history across every run is the same bug wearing a different hat.

With identity fixed, three charts carry real signal:

Publish them where they are already looked at — a Grafana panel fed from your results JSON, the report's own trend widgets, or a Slack/Teams digest posted after each nightly — because a chart nobody navigates to is not a signal. Feed the flaky count from the retry data you started recording in Parallelization and Flaky Test Management: the report and the quarantine list should agree, and if the dashboard says 12 flaky while the list holds 4, one of them is fiction.

Two measurement traps to avoid. Duration trends need the same hardware and shard count to be comparable — a move from 4 workers to 8 halves wall-clock time without a single test getting faster, so annotate the chart at configuration changes rather than letting the team credit an optimization that never happened. And pass rate can be gamed by deletion: a suite that drops its hardest tests climbs to 100% while real coverage falls, which is why the trend must sit next to test-count and quarantine-health metrics — three numbers together are much harder to flatter individually.

Retention is a policy decision as well: keep a fixed window of history (Allure carries the history/ directory forward between runs, and --keep-history preserves it on generate), expire raw results after a couple of weeks, and accept that a visible gap after a server migration is honest — a chart silently recomputed from partial data is not. When a metric moves, the first question is always "did the measurement change?", which is why worker counts, retry settings, and runner-image bumps get annotated on the chart at the moment they land. A trend nobody can explain becomes a trend nobody trusts, and an untrusted chart is worse than no chart at all, because people keep making decisions with it anyway.

The same run has to serve two readers with nothing in common: an engineer asking which assertion, on which build, with what evidence? and a product lead asking can we ship, and what is the risk? Build both from one dataset, but do not show the same screen to both — an engineer given an executive summary and a stakeholder given a failure log each waste their time.

For triage, evidence must be attached to the failed test, not stored three clicks away.

// playwright + allure: attach at the moment of failure
await test.info().attach("response.json", {
  body: JSON.stringify(await res.json(), null, 2),
  contentType: "application/json",
});
await test.info().attach("screenshot", { path: testInfo.outputPath("failure.png") });
await test.info().attach("console.log", { body: browserLogs.join("\n"), contentType: "text/plain" });

The default set for a failed test is screenshot, video, console/browser logs, the request and response pair, and the seed data or fixture ids — enough that triage starts in the report instead of in a local reproduction attempt. For device-farm runs, attach the vendor session URL rather than the video blob (the pattern from Device Farms in CI). Then add categories, which turn a wall of red into classified work:

[
  { "name": "Product defect", "matchedStatuses": ["failed"], "messageRegex": ".*expected.*" },
  { "name": "Test bug", "messageRegex": ".*TimeoutError.*" },
  { "name": "Infrastructure", "messageRegex": ".*device.*unavailable.*" },
  { "name": "Known issue", "flaky": true, "expires": "2026-12-01" }
]

For stakeholders, publish status, not logs. What belongs on the shared dashboard is: open/blocking failures against the release candidate, trend of pass rate and flaky rate, coverage or gate status, quarantine health, and the top five risks with an owner each — at a stable URL that is always current, which is the real answer to "communicate quality without meetings". The reporting principle that keeps it honest: never present a raw failure count without classification, because three failures that are all known environment issues should not read worse than one new defect, and a green suite hiding 40 quarantined tests should not read better than it is. Label the data by build and by environment, stamp "as of " on every view, and route questions about a number to the drill-down page rather than to a person.

Done well, this closes the loop the whole phase has been building: the pipeline gates the merge, the gate publishes numbers, the numbers carry evidence, and the dashboard shows the same numbers to everyone — one dataset, one story, no meeting required.

Wire the report into the flow rather than hoping it gets consulted: post the report URL with the classified summary as a PR comment when checks fail, put the failing test's deep link in the CI annotation, and include it in the nightly message. Every link must be deep — straight to that test's history page, not the report's front door — because the extra click is where triage goes to die. Then close the loop with a classification step: the engineer sees the evidence, labels it defect / test bug / infrastructure, and the label feeds the categories in the next run — so the "known issues" bucket is maintained by whoever triaged, instead of by a separate reporting chore nobody schedules.

The next topic picks up the first number most teams measure after functional correctness: Performance Concepts and Baselines — how fast the system is.