Wiring the Capstone into CI

A framework that runs only when someone remembers to run it is a hobby. The capstone earns its name when two triggers make two different promises: every pull request gets a fast verdict on the critical path, and every night the full suite runs whether anybody asked for it or not.

# .github/workflows/capstone.yml
name: capstone
on:
  pull_request:
    branches: [main]
  schedule:
    - cron: '17 2 * * *'      # 02:17 UTC nightly — avoid the :00 stampede
  workflow_dispatch:          # manual runs during an incident

concurrency:
  group: capstone-${{ github.ref }}
  cancel-in-progress: true    # a new push supersedes the in-flight run

jobs:
  smoke:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    timeout-minutes: 12
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npm run test:smoke -- --reporter=github

  full:
    if: github.event_name != 'pull_request'
    runs-on: ubuntu-latest
    timeout-minutes: 90
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npx playwright install --with-deps
      - run: npm run test:full

The split is a cost decision, not a convenience. Your full suite is 1,840 tests at roughly 38 minutes across four shards; running that on every push means a developer waits 38 minutes for a verdict and the repository burns runner minutes on revisions that get superseded within the hour. The smoke lane is 12 tests tagged @smoke — login, search, add-to-cart, pay, order-visible — finishing in about 3 minutes on one worker. The PR gate buys speed by accepting risk; the nightly run pays that risk back daily. Treat the 12-test list as a budget: adding a thirteenth test requires a discussion about what leaves.

Three details decide whether the schedule survives contact with reality. GitHub disables scheduled workflows after 60 days of repository inactivity — a capstone nobody commits to quietly stops testing itself, so keep workflow_dispatch available and either commit regularly or add a weekly heartbeat job. Cron executes in UTC, so 17 2 * * * lands at 8:17 PM EST for a US team; state the intended local time in a comment next to it. And concurrency with cancel-in-progress reclaims real compute: when review feedback produces three pushes in ten minutes, two runs are cancelled instead of queued, which is also why the checks a reviewer sees always belong to the newest commit.

One more contract: what a PR run proves about the merge, not just the branch. With strict: true on the required checks (the mechanism Integrating Tests into CI covers), the smoke lane re-runs against current main before merge. If your smoke coverage is thin, add a push: branches: [main] trigger to the full job so the merged result is verified before it can ship, and let nightly failures open issues automatically via actions/github-script rather than waiting for a human to notice a red badge.

The environment the smoke lane tests is its own decision, and it deserves a comment in the workflow file. Either point it at shared staging (env: BASE_URL: ${{ vars.STAGING_URL }}) and accept that parallel pull requests share state — per-run fixture data from Designing the Capstone Framework exists precisely to contain that — or deploy a per-PR preview environment and test that, which isolates runs at the cost of deployment time inside the 12-minute budget. What you cannot do is leave it implicit: a smoke lane aimed at "whatever was last deployed to staging" is not a gate, it is a dice roll with a green badge.

Designing the Capstone Framework decided what runs and where it lives; this decided when it runs and what a run costs. The next piece is the part everyone actually feels: what happens when a run goes red.

A red run that changes nothing is reporting. A quality gate changes what the team is allowed to do — and the only thing that changes behavior is a non-zero exit code attached to a check the branch requires. Three gate types carry the capstone, each with a different blast radius.

The first gate is coverage, enforced by the runner itself so it cannot be talked out of it:

// jest.config.cjs
module.exports = {
  collectCoverageFrom: ['src/**/*.ts', '!src/**/*.d.ts'],
  coverageThreshold: {
    global: { branches: 80, functions: 82, lines: 80, statements: 80 },
    './src/orders/**': { lines: 90, branches: 85 },  // money paths demand more
  },
};

Push a commit that drops order-lines coverage and the job ends with ERROR: Coverage for lines (78.4%) does not meet global threshold (80%), exit code 1. The per-directory override is the interesting part: a single global number lets critical code rot behind well-tested utilities. Money paths at 90% and everything else at 80% encodes where the risk actually is.

The second gate is the test run itself, and its settings belong in the config file rather than in someone's shell alias:

// playwright.config.ts
export default defineConfig({
  forbidOnly: !!process.env.CI,      // a focused .only must fail CI, not vanish
  retries: process.env.CI ? 2 : 0,   // absorb infra blips; local stays strict
  reporter: process.env.CI ? [['github'], ['blob', { uploadToServer: false }]] : ['html'],
});

Retries deserve precision because they are the most abused setting in the industry. A test that fails once and passes on retry is still flaky — Playwright counts it as flaky, and your gate must surface flaky counts even when the run is green. The policy that keeps retries honest: retries: 2 absorbs infrastructure blips, a test marked flaky in 5 of its last 30 runs goes to the @quarantine tag (excluded from the gate via --grep-invert @quarantine, still executed nightly and reported), and anything failing three consecutive runs outright is a product bug until proven otherwise.

The third gate is the platform-side enforcement, which is where "we have gates" becomes "you cannot merge":

# repository ruleset on main
ruleset:
  required_checks: [capstone-smoke, coverage-gate, lint]
  strict: true            # head must be current with main for checks to count
  enforce_admins: true    # yes, including the person who wrote the framework

Each gate needs an owner and a stated failure reaction, or it decays into noise:

That table is the answer to the classic interview question "how do you balance quality gates with velocity?" — you do not block on everything; you block on what protects the critical path, and everything else becomes a tracked obligation with a deadline. Blocking merges on nightly-only failures would stall the team for environment problems; treating smoke failures as advisory would make the gate decoration. The boundary is deliberate, documented, and revisited in the retrospective from Maintenance, Review, and Demo. Before those gates can judge anything, though, the failures have to arrive with their evidence — which is the next problem: artifacts, traces, and a suite that fits the budget.

Your suite runs 1,840 tests in 38 minutes on a single runner. Product reviewers abandon the PR process within a week. The fix has two halves — make the suite faster with sharding, and make every failure self-investigating with artifacts, because a fast red build without evidence just moves the debugging onto someone's laptop.

Sharding splits the suite across parallel runners. Playwright's native support is one flag per runner:

jobs:
  e2e:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    strategy:
      fail-fast: false        # one red shard must not cancel the others
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npx playwright install --with-deps
      - run: npx playwright test --shard=${{ matrix.shard }}/4
        env: { BASE_URL: https://staging.acme.dev }
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: shard-${{ matrix.shard }}
          path: test-results/
          retention-days: 14

  report:
    needs: e2e
    if: always()
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
        with: { path: blobs, merge-multiple: true }
      - run: npx playwright merge-reports --reporter html blob
      - uses: actions/upload-artifact@v4
        with: { name: html-report, path: playwright-report/ }

The measurements matter more than the flags. One runner: 38 minutes. Four shards: the longest shard settles near 11 minutes, because Playwright splits by test file and a 4-minute checkout.spec.ts cannot be subdivided — sharding scales only as evenly as your fattest file, which is a permanent argument for keeping specs small. Add the ~90-second overhead for install and report merging and you land around 12:30 wall-clock, inside a stated 15-minute PR budget. fail-fast: false is not optional: with the default, one red shard cancels the rest and you lose the full failure picture you paid runners for.

One v4-specific trap bites every team exactly once: artifact names must be unique within a run, so a bare name: test-results in every shard dies with 409 Conflict — an artifact with this name already exists on this workflow run. The shard-${{ matrix.shard }} suffix above is the fix, paired with merge-multiple: true on download so the report job sees one flat directory instead of four nested ones.

Evidence is the second half, configured in the framework and published by the pipeline:

// playwright.config.ts — the evidence policy
use: {
  trace: 'retain-on-failure',    // full DOM/network recording for red tests only
  screenshot: 'only-on-failure',
  video: 'retain-on-failure',
}

and published with the fail-after-publishing pattern: upload steps run if: always(), then the job exits non-zero. Exit first and the artifacts die with the job. Sizes are manageable precisely because retention is conditional — green runs store nothing, a red run stores roughly 8–40 MB per failing test, and retention-days: 14 keeps the repository's artifact storage bounded instead of accepting GitHub's 90-day default for trace zips nobody will open again.

Finally, measure the pipeline itself, not just the product. Write a timing table to $GITHUB_STEP_SUMMARY after every run:

{
  echo "| Job | Duration | Budget |"
  echo "|---|---|---|"
  echo "| smoke | ${{ job.steps.smoke.outputs.duration }}s | 180s |"
  echo "| e2e shard p95 | 10:42 | 15:00 |"
} >> "$GITHUB_STEP_SUMMARY"

A budget you do not measure drifts: 12 minutes becomes 19, then the smoke lane quietly grows past 20 tests and the PR gate dies the death of "I'll just run it locally". When the number crosses the budget, the levers in order are: trim the smoke set to the true critical path, move setup work from UI clicks to API fixtures (the trick from Designing the Capstone Framework), and only then add shards — each one is a monthly bill. With green runs fast and red runs self-evidencing, the data is rich enough to be worth publishing, which is what Reporting for Stakeholders does with it.