Offline Evaluation Harness

A developer rewords one sentence in a system prompt — "be concise" becomes "answer in at most two sentences" — and the change passes every manual check. Two weeks later, a customer reports that the assistant has started refusing to answer multi-part questions. The reworded instruction did exactly what it said; it just also did something nobody checked.

The team had no way to know. There was no fixed set of inputs to run, no recorded scores to compare against, and no gate between a prompt edit and production. Every change was validated by the same method: try it, see that it looks fine, ship it.

An offline evaluation harness closes that loop. It is a program that runs a fixed dataset through the system under test, scores every result, and aggregates the metrics — on every change, before merge. The dataset comes from Building Eval Datasets; the scorer can be exact match, a rubric, or an LLM judge. The harness is the machinery that turns those artifacts into a gate.

A harness has four stages: load the dataset, run each case through the system, score each output, aggregate. The whole thing fits in a page, and every line earns its place.

import json
from pathlib import Path

def run_harness(dataset_path: str, run_case, score_case) -> dict:
    dataset = json.loads(Path(dataset_path).read_text(encoding="utf-8"))
    results = []
    for case in dataset:
        output = run_case(case)          # call your app: prompt + model + tools
        score = score_case(case, output)  # exact match, rubric, or judge
        results.append({"id": case["id"], "output": output, "score": score})

    passed = sum(r["score"] for r in results)
    return {
        "pass_rate": passed / len(results),
        "results": results,             # per-example, not just the average
    }

Input: the 120-case regression dataset, a new prompt version.

Output: pass_rate 0.84 — down from 0.91 on the previous run, with 11 failing case ids listed.

Three decisions are baked into those ten lines:

• The dataset is loaded from disk, never rebuilt. The harness measures the system against a fixed bar; a dataset that changes between runs makes score changes uninterpretable.
• run_case is a seam, not an implementation. It can call a prompt template, a full RAG pipeline, or an agent loop. The harness does not care what the system is — only that it takes a case and returns an output.
• Per-example results are kept, not just the aggregate. An average can stay flat while ten cases regress and ten new ones pass; the per-example list is what makes the next chapter possible.

The scorer is a function too. Deterministic tasks score with exact match or schema validation; open-ended generation scores with a rubric or judge. One harness, many scorers — the scoring layer is swappable without touching the run loop.

A pass rate is a summary. The artifact a developer actually needs is the list of cases that changed — the diff between the last run and this one.

def diff_runs(baseline: dict, current: dict) -> list[dict]:
    before = {r["id"]: r["score"] for r in baseline["results"]}
    after = {r["id"]: r["score"] for r in current["results"]}
    return [
        {"id": cid, "before": before[cid], "after": after[cid]}
        for cid in before
        if before[cid] != after[cid]
    ]

Input: baseline run with pass_rate 0.91, current run with pass_rate 0.84.

Output:

regressions:  extract-invoice-003  1 -> 0
              summarize-ticket-014  1 -> 0
              refund-policy-007    1 -> 0
improvements: shipping-eta-011    0 -> 1

The diff is the review artifact. A developer reading "pass rate dropped 7 points" still has to find what broke; reading the diff, the regression is already isolated to three named cases with their inputs and outputs in the run log. The improvements column matters too — a change that fixes three cases and breaks three is a trade to discuss, not a regression to revert.

Failure modes of diffing done badly:

• Diffing only the aggregate. The average hides compensating changes; the per-example diff is the only honest view.
• No stored baseline. A diff needs two runs. The harness writes every run to disk with its commit hash, prompt version, and model string, so any run can be diffed against any other.
• Diffing against a moving dataset. If cases were added between runs, the diff mixes dataset changes with system changes. The dataset version is recorded alongside the run.

A harness that runs when someone remembers to run it is not a gate. The gate is CI: any change to prompts, models, or retrieval config runs the regression split, and the build fails when the score drops.

# .github/workflows/eval.yml
on:
  pull_request:
    paths: ["prompts/**", "src/llm/**", "evals/**"]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python -m evals.run --suite regression --compare main
      - run: python -m evals.gate --min-pass-rate 0.90 --max-regressions 0

The gate has two thresholds, and the second is the one teams forget: max-regressions 0 fails the build even when the aggregate passes, because a change that fixes easy cases and breaks hard ones is a regression, not a wash.

Running the full dataset on every prompt edit is slow and expensive, so development runs cache provider responses:

import hashlib, json
from pathlib import Path

CACHE = Path(".eval_cache")

def cached_call(key: str, fn):
    CACHE.mkdir(exist_ok=True)
    path = CACHE / f"{hashlib.sha256(key.encode()).hexdigest()}.json"
    if path.exists():
        return json.loads(path.read_text(encoding="utf-8"))   # replay, $0
    value = fn()
    path.write_text(json.dumps(value), encoding="utf-8")
    return value

Input: the same 120-case suite run three times during a prompt edit session.

Output: first run 120 API calls, next two runs 0 — the cache key includes the prompt hash, so editing the prompt misses the cache and re-runs only what changed.

The cache key must include everything that affects the output: prompt version, model string, temperature. A cache that misses those serves stale responses and the harness measures a system that no longer exists. CI runs with a cold cache; local development runs warm.

Offline gates catch regressions before merge. They cannot catch what only appears under real traffic — that gap is what online metrics exist to close.