Regression Suites for Model Changes

The dashboard shows a slow slide: invoice extraction accuracy drops from 94% to 86% over three weeks. No code shipped. No prompt changed. The model string in the config is identical. What changed is that the provider behind that string changed — a silent rollout of new weights behind the same alias, the kind of update that arrives with no changelog and no breaking-change notice.

This is the specific failure a regression suite exists for. Your own changes are visible in git review; upstream changes are invisible until the metrics move. A pinned regression suite detects drift fast because it does not depend on anyone noticing: the same golden cases, run on a schedule, scored the same way, with any deviation from the frozen expectation treated as an incident.

The suite is a slice of the eval dataset with one difference: the expectations are frozen and the discipline around changing them is strict. Where the eval dataset grows and adapts, the regression suite is a contract with the behavior you have decided is known-good.

A golden case is a pinned input with a frozen expectation. What the expectation looks like depends on whether the task is deterministic:

GOLDEN_CASES = [
    {
        "id": "extract-invoice-001",
        "input": "Invoice #INV-2041, total $142.50, due 2026-03-01",
        "expected": {"invoice_id": "INV-2041", "total": 142.50, "due": "2026-03-01"},
        "mode": "exact",          # deterministic task: freeze the output
    },
    {
        "id": "summarize-ticket-014",
        "input": "Customer reports duplicate charge...",
        "expected_range": {"rouge_l": (0.55, 1.0), "judge_min": 0.8},
        "mode": "range",          # open-ended: score within a band
    },
]

def check_golden(case: dict, output: dict) -> bool:
    if case["mode"] == "exact":
        return output == case["expected"]
    return (case["expected_range"]["rouge_l"][0] <= output["rouge_l"]
            and output["judge_score"] >= case["expected_range"]["judge_min"])

Input: the extraction case against a model that returns {"invoice_id": "INV-2041", "total": 142.5}.

Output: False — the dropped trailing zero is a real regression, not a formatting difference.

The two modes are not stylistic. Exact equality on a non-deterministic task produces false failures: the same input legitimately yields different wording on every run, and a suite that cries wolf weekly is a suite that gets disabled. Open-ended generation gets a score range — a floor on a similarity metric plus a judge threshold — wide enough to absorb legitimate variation, narrow enough to catch real degradation. The judge component is the one from LLM-as-Judge Techniques, calibrated against human labels before the range is trusted.

The suite runs on a schedule — daily or on every provider alias change — even when nothing on your side changed. Upstream models change anyway, and a suite that only runs when you change something can only detect your own mistakes.

provider model update (silent)
        │
        ▼
scheduled suite run ──► regression detected
        │
        ▼
investigate: prompt? data? model drift?
        │
   ┌────┴────┐
   │         │
fix it   re-baseline (only with a written cause)

The investigation step is where the discipline lives. A regression means the system's behavior on a frozen case changed, and there are exactly three causes: your code changed, your data changed, or the upstream model changed. Each has a different fix, and re-baselining without identifying the cause converts all three into "the new normal" — the damage is absorbed into the baseline and never repaired.

Re-baselining rules that keep the suite honest:

• Every re-baseline requires a linked investigation note naming the cause.
• A re-baseline that lowers an expectation is reviewed like a code change, by someone other than the author.
• Re-baselining more than a handful of cases at once is a signal the suite is wrong, not the model.
• The previous expectation is kept in history, so a re-baseline can be reverted.

The suite runs inside the same harness as the rest of the eval dataset — the regression suite is the harness's always-on, never-retuned slice.

A regression suite is not the whole eval dataset. It is the cases whose behavior you have decided must never change, selected deliberately:

The last two rows are what make the suite a living memory. Every production incident adds a case — the same promote-failure loop as the eval dataset — so the suite accumulates the system's entire history of known failures. A provider update that re-breaks a fixed bug is caught in minutes, not discovered by a customer.

Failure modes of suite maintenance:

• Re-baselining without cause. The baseline drifts downward until the suite asserts nothing.
• A suite too small to detect drift. Twenty cases cannot see a 5% degradation; the suite needs enough cases that a real regression is statistically visible.
• Only happy-path cases. The suite passes while the long tail rots.
• Never scheduling runs. A suite that runs on demand detects only the changes someone remembered to check for.

When the scheduled run fires, the corrections it surfaces are exactly what human review workflows exist to adjudicate — the loop closes from detection to labeled fix.