Integrating Tests into CI

A suite that only runs on a developer's laptop is a suggestion; a suite that runs on every pull request and can block the merge is a process. The difference between "we have tests" and "our tests matter" is entirely in the pipeline wiring, and the wiring has three decisions: when it runs, what runs first, and what happens to a failure. A remote suite from Device Farms in CI is nothing special here — it is another job in the same file, with the same exit-code contract.

# .github/workflows/ci.yml
name: ci
on:
  pull_request:            # every PR, every push to it
    branches: [main]
  push:
    branches: [main]       # and re-verify the merged result

jobs:
  lint-and-unit:           # fast lane — target under 3 minutes
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20, cache: npm }
      - run: npm ci
      - run: npm run lint
      - run: npm test -- --ci --coverage

  integration:             # slow lane — only if the fast lane passed
    needs: lint-and-unit
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20, cache: npm }
      - run: npm ci
      - run: docker compose -f ci/deps.yml up -d   # db, queue
      - run: npm run test:integration
      - uses: actions/upload-artifact@v4
        if: failure()
        with: { name: test-logs, path: logs/ }

The ordering is the point. Fast lane first — lint, unit, and a 30-second smoke — because the value of a check decays with latency: a failure reported in 90 seconds is fixed while the context is still loaded; the same failure at 25 minutes arrives after the author has moved to another task, and they context-switch back at roughly double the cost. needs: encodes that dependency so a broken unit test does not burn five minutes of integration environment first. The same ordering logic reappears one level later, when tests gate a deployment rather than a merge — see DevOps Automation Principles.

Every job needs an explicit timeout (timeout-minutes: 20), because a hung test without one holds a runner for the platform maximum — six hours on GitHub — and the next PR queues behind it. And trigger on both pull_request and push to main: PRs verify the change against its base, while the push run verifies the merged result, catching conflicts that only appear after integration. The same shape exists in every CI system — only the syntax moves. GitLab CI expresses the ordering as stages plus needs in .gitlab-ci.yml with rules: - if: $CI_MERGE_REQUEST_IID; Jenkins gets it from a declarative pipeline { stages { stage('fast') ... } } with when { changeRequest() }. Three properties must hold in any of them. Run on the merge request, not only on a nightly schedule — a defect found 14 hours later costs a context switch. Load secrets from the CI's secret store, never from the repository, so a log line cannot leak them. And cancel superseded runs: GitHub's concurrency: { group: "ci-${{ github.head_ref }}", cancel-in-progress: true } drops the run for a commit that has already been replaced, which keeps busy branches from queueing five obsolete jobs. Add explicit least-privilege permissions: { contents: read } while you are there — a workflow with a default write token and a compromised third-party action is how a test pipeline becomes a supply-chain incident.

The last piece is what a failure does — that is the third chapter — but first there is time to reclaim with caching.

A representative Node pipeline: npm ci takes 78 seconds, Playwright downloads three browsers at 110 seconds, and the suite itself runs 95 seconds. The tests are the shortest part of the build. Caching targets the other two, and the arithmetic is why it is worth configuring properly rather than accepting defaults.

Dependency caching works because package managers already have a content-addressed cache — CI's job is to persist it between runs. In GitHub Actions the setup step does it in one line, and the key must include the lockfile hash:

- uses: actions/setup-node@v4
  with:
    node-version: 20
    cache: npm                  # persists ~/.npm keyed on package-lock.json
- run: npm ci                   # installs from the cache; still a clean node_modules

Note what npm ci still guarantees: it deletes node_modules and installs exactly what the lockfile says, so reproducibility is untouched — what gets cached is the download cache, not the tree. The lockfile in the key is the correctness rule: change a dependency, get a new key, get a fresh cache; omit it and a stale or mismatched cache produces the worst class of CI bug, one that passes in CI and fails locally (or the reverse).

Browser binaries are the bigger prize because they never fit in a package manager's cache:

- name: Cache Playwright browsers
  uses: actions/cache@v4
  with:
    path: ~/.cache/ms-playwright
    key: playwright-${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}
    restore-keys: playwright-${{ runner.os }}-
- run: npx playwright install --with-deps chromium

The restore-keys fallback matters: when the lockfile changes but Playwright's version did not, the prefix match restores the previous cache and the install step verifies in a second or two instead of re-downloading 130 MB. Mirror the pattern wherever binaries live — ~/.gradle/caches for Java (keyed by **/*.gradle*), ~/.m2/repository, pip's cache, Docker layers via docker/build-push-action with cache-from/cache-to, and the Cypress binary in ~/.Cypress. Measured effect on that Node pipeline: cold 4:23 → warm 2:11, and on Gradle-heavy JVM projects it is common to see 6+ minutes fall under 2.

Two caching failure modes to recognize in a code review: a key that never changes (no lockfile in it) invites the stale-cache bug above, and a cache that is too broad (caching node_modules itself across lockfile changes) delivers corrupt trees that only some runners see. Cache downloads, key them by inputs, and let the package manager do the installing.

Measure the cache rather than assuming it: the actions/cache step prints Cache restored from key ... versus Cache not found, and p50 job duration before and after is the number worth publishing to the team. A cache that always misses usually has a volatile key (a timestamp, a runner-specific path, an unpinned actions/cache major version) or two jobs in a monorepo fighting over one key with different lockfiles. And keep cached content pure: many setups restore caches into the workspace of untrusted pull requests, so never cache secrets, .env files, or build output containing credentials — a cache is shared state, and shared state is precisely what makes parallel runs flaky.

With the heavy downloads gone, the pipeline's duration approaches the suite's real runtime, which is the number your feedback budget should be measured against.

The most dangerous line in any pipeline is npm test || true. It converts a failed suite into a green build with a red log nobody reads, and it is almost always added under time pressure with the best of intentions. The rule that makes CI worth having: a test error must produce a non-zero exit code that fails the job, and the job must be required for merge.

Exit-code plumbing is simple and worth being precise about:

npm test -- --ci                 # non-zero on failure -> job fails
npx jest --ci --maxWorkers=4 || EXIT=$?   # capture instead of swallow
if [ "${EXIT:-0}" -ne 0 ]; then
  echo "::error::unit tests failed — see the report artifact"
  exit 1                          # still fail, but after publishing artifacts
fi

The pattern behind that snippet is fail after publishing: upload the report (if: always() / if: failure() on the artifact step) before the job exits non-zero, or the only artifact you needed is the one you lost. The same applies to the continue-on-error and allowFailure flags — they belong to genuinely optional jobs (a best-effort preview deployment, an advisory linter), never to a test suite; a test job marked continue-on-error is a warning wearing a required check's name.

Exit codes only enforce themselves once the repository does. In GitHub, mark the checks as required status checks under branch protection (or rulesets):

# repository ruleset: protection for main
required_status_checks:
  contexts: ["lint-and-unit", "integration", "contract-gate"]
strict: true                 # branch must be up to date before merging
checks:
  - enforcing: true

With that in place, a red PR cannot merge through the normal path — no reviewer discipline needed, no "merge and fix it later". strict: true adds the second half: the PR is tested against the current main, not the one it branched from three weeks ago, which is where integration surprises hide.

Keep the taxonomy honest, because "the build failed" has three different owners: test failures (the code is wrong — block), infra failures (the device farm lost a session, the registry timed out — retry once, do not blame the author), and flake (a test that fails intermittently — visible, counted, and owned, never silently retried into green). Conflating the three is how teams learn to distrust CI and start clicking "re-run" reflexively — and keeping them apart is exactly what Quality Gates and Merge Policy turns into written, machine-checked policy.

Two details decide whether "required" really means required. Enable enforcement for administrators too (rulesets can), otherwise a break-glass merge outside the audit log is one click away — and an unaudited bypass is how the check acquires an asterisk nobody documented. Name checks stably: list lint-and-unit, not lint-and-unit (ubuntu-22.04, node 20), because when the runner image or runtime is bumped, a name-derived check silently disappears from the required list and the gate quietly stops gating — the failure mode that leaves teams wondering why a red PR merged last Thursday. Both are configuration, and both are invisible until someone audits why a blocked change reached main.

Wire the trigger, order the lanes, cache the heavy parts, and let the exit code — not a person's attention — decide whether the merge happens.