Testing AI Code

The team's test suite called the model directly. Three edits to the system prompt cost ninety seconds of wall time, a bill line that grew with every commit, and a red build whenever the provider rate-limited CI — failures that said nothing about the code. The prompt logic was testable offline; only the transport needed the network.

Replace the one function that touches the wire

# summarizer/client.py
import httpx, os

URL = "https://api.openai.com/v1/chat/completions"
HEADERS = {"Authorization": "Bearer " + os.environ["OPENAI_API_KEY"]}

def complete(prompt: str, *, timeout: float = 60.0) -> dict:
    r = httpx.post(URL, headers=HEADERS, json={"model": "gpt-4.1-mini",
                                               "messages": [{"role": "user", "content": prompt}]},
                   timeout=timeout)
    r.raise_for_status()
    return r.json()

# summarizer/pipeline.py
from dataclasses import dataclass
from summarizer import client

class TruncatedAnswer(RuntimeError):
    pass

@dataclass
class Summary:
    text: str
    tokens: int

def summarize(document: str) -> Summary:
    data = client.complete(f"Summarize: {document}")     # client.complete resolved at call time
    choice = data["choices"][0]
    if choice["finish_reason"] != "stop":
        raise TruncatedAnswer(choice["finish_reason"])
    return Summary(text=choice["message"]["content"],
                   tokens=data["usage"]["total_tokens"])

# tests/test_summarizer.py
def test_extracts_answer_and_tokens(monkeypatch):
    fake = {
        "choices": [{"message": {"content": "Short summary."},
                     "finish_reason": "stop"}],
        "usage": {"total_tokens": 52},
    }
    monkeypatch.setattr("summarizer.client.complete", lambda prompt, **kw: fake)

    result = summarize("Some long document text")
    assert result.text == "Short summary."
    assert result.tokens == 52

Output:

tests/test_summarizer.py ..                                      [100%]
2 passed in 0.06s

What the test kept and what it gave up:

• Kept: the real summarize body, the extraction logic, the shape assertions. A wrong index into choices still fails here.
• Gave up: the HTTP call, the key, the provider's uptime. That surface is re-verified by the nightly smoke tier at the end of this topic, not on every commit.

monkeypatch.setattr takes a dotted path, and it must be the name resolved at call time — pipeline.py calls client.complete(...), so the attribute on the summarizer.client module object is the one to swap. Had the module done from .client import complete, the lookup would happen in summarizer.pipeline instead, and the patch must follow it there. Getting this wrong silently leaves the real request in place: the test passes, slowly, on the CI credit card.

A fixture is the right home when several tests share one fake:

import pytest

@pytest.fixture
def fake_completion():
    return {"choices": [{"message": {"content": "stub"},
                         "finish_reason": "stop"}]}

def test_truncated_answer_raises(fake_completion, monkeypatch):
    fake_completion["choices"][0]["finish_reason"] = "length"
    monkeypatch.setattr("summarizer.client.complete",
                        lambda *a, **k: fake_completion)
    with pytest.raises(TruncatedAnswer):
        summarize("Some long document text")

Assert on failures too. A fake that only ever returns the happy path cannot detect the day the truncation branch stops raising.

Field-level tests confirmed every value was correct, and the rendered digest still changed on a Tuesday: the bullet list gained an extra space, the dates flipped locale, and nobody caught it because no assertion ever looked at the final string. Correct fields do not imply correct formatting — only a test that compares the whole output can see that drift.

Store the expected output, compare normalized forms

# tests/data/summary.golden.txt
Quarterly latency dropped after the retry change.

- p95 down 18%
- tokens: 52

def test_render_matches_golden(golden, monkeypatch):
    monkeypatch.setattr("summarizer.client.complete",
                        lambda *a, **k: fake_completion())
    rendered = render(summarize(DOC))
    assert rendered.strip(), "empty output must never match an empty golden"
    assert normalize(rendered) == golden.read_text()

The normalize step is the whole trick — goldens break the moment output contains something that legitimately varies per run:

import re

def normalize(text: str) -> str:
    text = re.sub(r"\d{4}-\d{2}-\d{2}", "<DATE>", text)   # dates vary by run day
    text = re.sub(r"req-[0-9a-f]{8}", "req-<ID>", text)   # ids vary per request
    return text.replace("\r\n", "\n").rstrip() + "\n"      # line endings, trailing space

Failure output:

E   - - tokens: 52
E   + - tokens:  52

The diff pinpoints the doubled space inside a bullet — invisible in field assertions, obvious here. Updating becomes an explicit act: pytest --update-golden rewrites the stored files, so the change lands in the pull request as a reviewable diff. The reviewer approves a formatting change because they can see it.

Three rules keep goldens from rotting:

1. Normalize only what genuinely varies. Every placeholder added is real drift you have chosen to stop seeing.
2. Never normalize the value under test. Testing date formatting while the normalizer replaces dates with <DATE> produces a test that cannot fail.
3. Fail on empty. An empty golden matching empty output is a green test asserting nothing — the assert rendered.strip() line above is what closes that hole.

The remaining gap is not formatting at all: mocks only ever produce strings someone thought to write down.

Freezing output catches formatting drift, and mocking catches broken logic — both only over inputs somebody remembered to write. The parser that passed six hand-written fixtures meets a production string where the JSON sits inside a quoted list with a brace in it, and no test in the suite had ever built one.

Property tests: let the generator find it

import json
from hypothesis import given, strategies as st

def extract_first_object(text: str) -> dict:            # the function under test
    start, end = text.find("{"), text.rfind("}")
    return json.loads(text[start:end + 1])

@given(st.recursive(
    st.none() | st.booleans() | st.integers() | st.text(),
    lambda children: st.lists(children) | st.dictionaries(st.text(), children),
    max_leaves=8,
))
def test_roundtrips_any_json(value):
    payload = json.dumps(value)
    assert extract_first_object("model said: " + payload + " done") == value

Output: 500 passing examples in 1.24s — half a thousand inputs the fixture list never imagined: nested lists, empty strings, unicode, values whose text contains braces.

Two properties make a test like this trustworthy. It must be reproducible — pin the hypothesis seed (or its database directory) in CI so a reported failure reruns exactly. And it must be meaningful: round-tripping the parser's contract survives refactors, while assertions on intermediate slicing would fail on any harmless rewrite.

Live smoke: the one test allowed to touch the network

import pytest, httpx
from summarizer.client import complete

@pytest.mark.live
def test_provider_still_accepts_our_request():
    data = complete("Reply with the word ok", timeout=30.0)
    assert data["choices"][0]["message"]["content"]

pytest -m "not live"    # every commit: offline, ~1s, free
pytest -m live          # nightly job: one request, seconds, costs tokens

It asserts almost nothing about your code — and that is exactly its job. It catches the failures offline tiers structurally cannot: a renamed endpoint, an auth scheme change, a model retired from the price list, a response envelope field removed.

What each tier is for

The offline tiers run on every commit precisely because they cost nothing. The live tier runs nightly precisely because it does not. Keeping that boundary explicit is what lets the whole suite stay fast enough that nobody is tempted to skip it — and what the provider's contract looks like from the other side is Your First LLM API Call.