OpenAPI Spec-Driven Testing

Your API has 47 operations. The hand-written smoke suite covers 12 of them, and endpoint number 43 shipped last sprint undocumented and untested because nobody remembered to add a test. The OpenAPI file already lists every path, every method, every parameter, and every documented status — so the fix is not more test writing, it is generating the smoke layer from the document so coverage grows with the spec instead of with someone's diligence.

# openapi.yaml (excerpt)
paths:
  /orders:
    get:
      operationId: listOrders
      parameters:
        - in: query
          name: limit
          schema: { type: integer, minimum: 1, maximum: 100 }
          example: 25
      responses:
        '200':
          description: OK
          content:
            application/json:
              schema: { $ref: '#/components/schemas/OrderPage' }
        '422': { $ref: '#/components/responses/ValidationError' }
  /orders/{id}:
    get:
      operationId: getOrder
      parameters:
        - in: path
          name: id
          required: true
          schema: { type: string, format: uuid }
      responses:
        '200': { description: OK }
        '404': { description: Not found }

A minimal harness walks paths × methods and asserts the two things a smoke layer can honestly claim — the route answers, and it answers with a documented status:

import { readFileSync } from "fs";
import YAML from "yaml";
const spec = YAML.parse(readFileSync("openapi.yaml", "utf8"));
const BASE = process.env.BASE_URL;

for (const [path, item] of Object.entries(spec.paths)) {
  for (const [method, op] of Object.entries(item)) {
    if (!/^(get|post|put|patch|delete)$/.test(method)) continue;
    const url = BASE + path.replace("{id}", "00000000-0000-4000-8000-000000000000");
    const res = await fetch(url, { method: method.toUpperCase() });
    const documented = Object.keys(op.responses || {});
    await expectSoft(documented.includes(String(res.status)),
      `${method.toUpperCase()} ${path} -> ${res.status} (documented: ${documented.join(", ")})`);
  }
}

A 404 or 405 on a documented operation is the signature failure: the spec promises a route the server does not serve. Do not hand-roll this when mature tools already do it — schemathesis generates fuzzed inputs from the same document and runs schema checks, and dredd replays documented transactions:

pip install schemathesis
schemathesis run --checks all --workers 8 https://staging.acme.dev/openapi.json
# 47 operations -> 47 checks, and a failure names operationId + path + check

Keep the layers honest about what they prove. Generated smoke proves existence and documented status only — it cannot know that GET /orders?limit=25 returns at most 25 items, that the list is sorted, or that totals add up. Those stay in your authored tests, the boundary datasets from Data-Driven API Testing, and business-rule assertions.

schemathesis run --checks all is worth decoding, because the check names state exactly what the generated layer asserts without any input from you: status_code_conformance (a response outside the documented set), negative_data_rejection (invalid input the spec says must be refused), missing_content_type, schema_conformance (body against its schema), and examples (each documented example round-trips). Failures are reported per operationId, so the log names the endpoint, the check, and the field in one line.

One caveat before someone finds it: a generated probe with no credentials may only ever exercise the unauthenticated path. Supply auth and seed real ids — a run that hits every route and collects 401 on all of them verified the router and nothing else. And if your team lives in Postman, importing openapi.yaml produces a collection with one folder per tag and documented parameters pre-filled: same document, different runner. What the generator gives you is the guarantee that no operation can ship without at least one automated probe, which is the question interviewers mean when they ask how you keep tests in sync with a large API.

Generating from the spec only proves the documented half. The other half is drift in the opposite direction: routes the server serves that the document does not describe, and response fields that exist at runtime but not in the schema. A one-way check misses both, so run the comparison both ways:

The route-set diff is a few lines because both sides reduce to METHOD path strings:

const METHODS = ["get", "post", "put", "patch", "delete"];
const documented = new Set(
  Object.entries(spec.paths).flatMap(([p, item]) =>
    METHODS.filter((m) => item[m]).map((m) => `${m.toUpperCase()} ${p}`))
);

const live = new Set(accessLog.map(({ method, route }) => `${method} ${route}`));

const undocumented = [...live].filter((r) => !documented.has(r));
const unimplemented = [...documented].filter((r) => !live.has(r));
console.log({ undocumented, unimplemented });

Feed live from something real rather than a hand-maintained list: a night's staging access log, your API gateway's route table, or the framework's own introspection (Spring Boot's /actuator/mappings, Express's app._router.stack). Running this as a nightly job against staging catches the endpoint someone added "just for the mobile team" without touching openapi.yaml.

The one detail that makes the live → spec direction usable is path normalization: access logs record /orders/6f1c0a2e while the spec declares /orders/{id}, so every real request looks undocumented until you template the ids. Normalize with the framework's own matcher (Spring's PathPattern, Express's path-to-regexp) or a regex that rewrites UUIDs, integers, and long hashes to {param} — and normalize both sets, so the log's 6f1c0a2e and the spec's {id} collapse to the same token. Expect a little legitimate noise: health probes (/healthz), static assets, and gateway-internal routes belong on an explicit allowlist so the report shows only real drift.

Split the cadence by cost: the route diff runs nightly against a staging access log, the field diff runs on every pull request against one captured response per operation. Drift has two speeds, and a nightly detector that is cheap and boring is exactly what keeps a report readable — the version that pages someone at 2 a.m. gets muted within a week.

The field-level diff uses the same trick one level down. Capture one representative response per operation — the spec's own example is a good baseline — and compare key sets against properties:

const declared = new Set(Object.keys(opSchema.properties || {}));
const actual = new Set(Object.keys(payload));
const extra = [...actual].filter((k) => !declared.has(k));    // undocumented field
const missing = [...declared].filter((k) => !actual.has(k));  // documented but absent

missing is the sharp one: a field the document promises and the server stopped sending is a silent contract break that survives every status-code assertion. extra is usually an announcement nobody made — legal to add, dangerous to discover by accident, and precisely the class of change that must appear in review rather than in production. If you already validate payloads against versioned schemas from JSON Schema Validation, wire this field diff to the same schemas so the two checks share one definition of "declared".

Every operation in the document can carry worked examples, and examples are the cheapest test data you will ever own — written by the same person who defined the contract, versioned in the same file, and impossible to let rot in a spreadsheet nobody opens.

components:
  schemas:
    Order:
      type: object
      required: [id, status, total]
      properties:
        id: { type: string, format: uuid, example: "6f1c0a2e-4b7d-4f21-9a3e-1c2d3e4f5a6b" }
        status: { type: string, enum: [pending, paid, shipped], example: "paid" }
        total: { type: number, example: 42.5 }

From there, two workflows pay off immediately. A mock server serves the examples before the backend exists, so frontend and contract tests start on day one and CI can run the suite without a live dependency:

npx @stoplight/prism-cli mock openapi.yaml -p 4010   # validates requests/responses against the spec
curl -s 'http://localhost:4010/orders?limit=25' | jq '.items[0].id'

Prism in proxy mode is the sharper tool for verification: point the suite at prism proxy openapi.yaml https://real-api.internal and Prism forwards traffic while checking both the request against the request schema and the response against the response schema — turning your existing tests into contract tests without rewriting an assertion.

Making the file the source of truth is a policy decision, and it needs mechanical enforcement because policies without gates decay within a quarter:

# .github/workflows/contract.yml
- name: Lint the spec
  run: npx @stoplight/spectral-cli lint openapi.yaml -r .spectral.yaml
- name: Fail on breaking changes
  run: npx oasdiff breaking main/openapi.yaml head/openapi.yaml

oasdiff compares the branch's document against the base branch and exits non-zero on breaking changes — a removed operation, a new required parameter, a field added to required — so the argument happens on the pull request, where it costs a comment, instead of after release, where it costs a client hotfix. Spectral catches what the diff cannot: missing operationId, undocumented error responses, examples that violate their own schema.

The rule to state plainly in review and in interviews: a pull request that changes an endpoint's behavior must change openapi.yaml in the same diff. Clients, mocks, generated SDKs, and the smoke suite all descend from that one file, so sync cost stays constant as consumers multiply — the alternative is O(consumers) hand-maintained copies.

Two downstream consumers of the same file are how you get organizational buy-in for the policy. Docs: npx @redocly/cli build-docs openapi.yaml publishes browsable, versioned documentation straight from the artifact, so the docs cannot lag the API without failing the same pipeline that gates the code. Codegen: openapi-generator-cli generate -i openapi.yaml -g typescript-fetch -o sdk/ hands every team a typed client, and that is a second enforcement mechanism — a field rename that slips past one reviewer now breaks their build the next time they regenerate. Source-of-truth status has a price: code-first teams that emit the spec as a build artifact must still run the diff gate, because generation cheerfully exports an undocumented route as undocumented. Whether the file is written by hand or produced from annotations, the invariant enforced in review is identical — behavior changed, file changed, same pull request. What the spec captures is shape; what it cannot capture is what each client expects of the provider, which is the gap Consumer-Driven Contract Testing fills.