Automated Accessibility Audits

qa-automated-a11y-audits-axe-ci -->

Manual keyboard passes do not run on every commit — that is the job for a scanner, and the scanner you want is axe-core, Deque's rule engine that powers most commercial audit tooling. The integration strategy is the same one you use for functional tests: run it where the page actually renders (a real browser), on every pull request, and fail the build on violations you have decided are unacceptable.

The strongest integration sits inside Playwright, because it scans the built, hydrated application — a critical point for single-page apps, where a static HTML lint sees an empty <div id="root"> and reports a perfectly accessible blank page:

import AxeBuilder from '@axe-core/playwright';
import { test, expect } from '@playwright/test';

test('shop page has no WCAG 2.1 AA violations', async ({ page }) => {
  await page.goto('/shop');
  const results = await new AxeBuilder({ page })
    .withTags(['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa'])
    .analyze();
  expect(results.violations, JSON.stringify(results.violations, null, 2)).toEqual([]);
});

Three options do most of the day-to-day work. include/exclude scopes the scan (new AxeBuilder({ page }).include('#checkout')) so a PR touching one flow does not fail on an unrelated legacy page. disableRules opts out of a single rule with a comment explaining why (disableRules: ['color-contrast']) — an explicit, reviewable exception, never a silent one. And withTags ties the run to a conformance target: wcag2a + wcag2aa + the wcag21* tags is the automated face of the "WCAG 2.1 AA" commitment from WCAG Accessibility Fundamentals.

Component-level scanning catches defects even earlier. For a React/Jest stack, jest-axe wraps the same engine around rendered markup:

import { axe, toHaveNoViolations } from 'jest-axe';
expect.extend(toHaveNoViolations);

it('renders an accessible login form', async () => {
  const { container } = render(<LoginForm />);
  expect(await axe(container)).toHaveNoViolations();
});

A failing component test names the file that broke, which beats a pipeline failure on a URL three components away. For whole-site coverage — marketing pages, docs, the flows no unit test touches — add Lighthouse CI and let its accessibility score act as a coarse gate:

# lighthouserc.yml
ci:
  assert:
    assertions:
      accessibility: ['error', { score: 0.95 }]
      'color-contrast': ['error', { score: 1 }]

Keep the score gate coarse (a single critical issue drops it hard) and let axe's per-rule output do the precise triage; teams that gate on 0.99 discover that a button-name violation and a minor region warning weigh the same to a percentage. Wire the run into CI with artifacts kept per PR — a JSON results file uploaded next to the Playwright report — so the fix loop starts from evidence, exactly as failures were handled in Test Reporting and Dashboards. Run the scan on PRs against changed routes, and schedule a nightly full crawl of authenticated flows (the axe DevTools extension is the manual cousin for exploring one page interactively).

Two determinism details decide whether the gate survives its first month. Freeze what varies: dismiss or stub the cookie banner, wait for the loading spinner to clear (await expect(page.getByRole('status')).toBeHidden()), and close auto-opening overlays — anything that mounts randomly produces violations that come and go, and flaky gates get disabled. And scan at a fixed viewport with animations disabled (reducedMotion: 'reduce' in Playwright), because a mid-transition text color reports a contrast violation that does not exist in the settled UI.

Then one fact keeps expectations honest, and it is the most important sentence in this topic: an all-green axe run does not mean the page is accessible.

qa-automated-a11y-audits-coverage -->

Studies of real-world audits keep landing on the same ratio: automated tooling detects roughly 30–35 % of accessibility defects. That is not a flaw in axe — it is the boundary between what can be decided from rendered markup and what requires human judgment. Knowing the boundary precisely is what makes you credible when someone proposes replacing manual testing with a scanner.

What the engine checks well is structural: information present in the DOM can be verified mechanically.

What no scanner can decide is everything about meaning and behavior. Alt text can be present and useless — alt="icon" passes the rule and communicates nothing; only a human knows the image's purpose. Focus order is a runtime sequence across the whole session, not a property of one element's markup. A custom date picker can carry a perfect role="button" and still be inoperable with a keyboard (2.1.1) — the attribute is right, the behavior is missing. Error messages, instructions, link text that makes sense out of context ("click here"), whether a heading hierarchy reflects the page's structure, whether a modal restores focus on close: all invisible to the engine.

The practical division of labor that results:

• Every PR: axe scan in the browser (structural regressions, contrast, names).
• Every new feature touching interaction: the keyboard-only pass from the previous topic — tab order, traps, Esc, focus visibility.
• Before release: a short screen-reader session (NVDA/VoiceOver) on the changed flows — you verify what is announced, not what is present.
• Periodically: a full manual audit against the criteria list, ideally by someone who has not built the feature.

Two technical caveats keep the automated half honest. First, scan the real rendered app with a real browser — hydration, lazy-loaded routes, and modal content that only mounts on open all defeat static analysis, so a scan of the initial HTML of a SPA is close to meaningless. Second, make the scan deterministic: animations mid-scan can flip computed contrast values, and dialogs or cookie banners that mount randomly produce violations that come and go — flaky accessibility gates get disabled within a month, which is worse than never having one.

With the coverage boundary drawn, the remaining question is organizational: what do you do about the hundreds of pre-existing violations the scan just surfaced? That is the last piece — the topic of the next chapter.

qa-automated-a11y-audits-debt -->

Turn the axe scan on for a mature product and the first run reports what looks like 400 violations. Ship a gate that fails on all of them and the team does one of two things: reverts the integration, or quietly stops running it. Neither outcome is better than no audit at all, so the policy needs as much design as the pipeline.

Start with impact, which axe assigns per violation: critical, serious, moderate, minor. The mapping to policy is a judgment call you should make explicitly and write down:

Then separate new breakage from existing debt with a ratchet baseline. Fingerprint each known violation (rule id + a stable selector path) and store the set in the repository; the gate fails only on fingerprints not in the file:

const results = await new AxeBuilder({ page }).withTags(WCAG_TAGS).analyze();
const current = results.violations
  .filter(v => ['critical', 'serious'].includes(v.impact))
  .flatMap(v => v.nodes.map(n => `${v.id}::${n.target.join(' ')}`));
const baseline = new Set(JSON.parse(fs.readFileSync('a11y-baseline.json', 'utf8')));

const regressions = current.filter(f => !baseline.has(f));
if (regressions.length) {
  throw new Error(`New accessibility violations:\n${regressions.join('\n')}`);
}
// The baseline may only shrink: deleting a fingerprint requires the fix, not a re-baseline

The ratchet has one essential property — the baseline can only shrink. Allow re-baselining on demand and the file becomes a dumping ground; require a linked fix ticket to remove an entry, and every sprint mechanically reduces the debt. Report the trend like any other quality metric (open violations by impact, top offending components) using the dashboard habits from Test Reporting and Dashboards, because a number nobody sees does not decay — it metastasizes.

Ownership turns the list into work. Each baseline entry carries a component owner, and the debt lives in the tracker with the same shape as any defect: rule id and criterion (color-contrast — fails 1.4.3), a screenshot or the rendered snippet, the failing page, and an impact. Severity decides the service level: critical debt fixed in the current sprint, serious debt scheduled within a quarter, moderate/minor parked deliberately. When a component has no owner, that is the finding — you cannot fix what nobody owns, and the audit's first deliverable is the org chart correction, not the CSS.

Interviewers ask how you introduce accessibility to a team that has none. The answer that lands: a scanning gate on new code from day one (no gate is too late to start), a ratchet so history never blocks a release, severity policy written down, and named owners — tooling creates the pressure, policy decides where it is applied. This whole workflow generalizes to the next class of quality gates, where the scans are looking for vulnerabilities instead of contrast ratios: OWASP Security Testing Basics.