Building Eval Datasets

A team ships a support assistant after testing it against twenty examples they wrote themselves. Every one passes. The first production week produces three tickets the team has never seen: a customer who pastes a whole invoice, one who asks about a policy that was renamed last quarter, and one who writes in a dialect of English none of the examples used. The assistant fails all three.

The failure is not the model. It is the dataset. Hand-written examples cluster around what the builder imagined users would say, and imagination is a narrow sample of reality. Real traffic arrives abbreviated, misspelled, off-topic, and phrased in the customer's vocabulary instead of the documentation's. A dataset built from invented examples measures the team's assumptions, not the product's behavior.

The fix is to treat the dataset as a sampling problem. Production logs already contain the inputs users actually send; your job is to draw a representative sample from them, label it, and keep it growing. The safety constraints from Bounding Agent Capabilities still apply — strip personal identifiers before anything enters the dataset — but the raw material is real traffic, not invented queries.

Three properties separate a dataset that measures from one that decorates:

• Representative: cases are sampled from production traffic and logged failures, stratified so rare intents are not drowned out.
• Labeled: every case carries a gold answer or rubric, so scoring does not depend on whoever happens to judge.
• Partitioned: regression, exploratory, and holdout splits keep tuning from overfitting to the set.

Sampling from logs is not SELECT * LIMIT 100. Raw logs are dominated by the most common intent, so a flat sample measures the easy majority and says nothing about the long tail where failures live. Stratified sampling fixes the mix; dedup keeps near-identical queries from inflating pass rates.

import hashlib
import json
import random
from collections import defaultdict

def content_hash(text: str) -> str:
    return hashlib.sha256(text.strip().lower().encode()).hexdigest()[:16]

def build_eval_set(log_path: str, per_stratum: int = 20) -> list[dict]:
    strata = defaultdict(list)
    with open(log_path, encoding="utf-8") as f:
        for line in f:
            rec = json.loads(line)
            strata[rec["intent"]].append(rec)   # bucket by behavior, not by user

    eval_set, seen = [], set()
    for intent, records in sorted(strata.items()):
        random.seed(42)                        # same sample on every rebuild
        for rec in random.sample(records, min(per_stratum, len(records))):
            h = content_hash(rec["query"])
            if h in seen:
                continue                       # near-duplicates skew pass rates
            seen.add(h)
            eval_set.append({
                "id": f"{intent}-{h[:8]}",
                "query": rec["query"],
                "gold": rec["gold_answer"],
                "stratum": intent,
            })
    return eval_set

Input: 5,000 logged sessions across 6 intents.

Output: 120 cases, 20 per intent, zero duplicate queries.

Three lines carry the design:

• strata[rec["intent"]] buckets by the behavior you care about. Sampling per bucket guarantees the rare "cancel my subscription" intent gets the same representation as "Where is my order," instead of being outnumbered fifty to one.
• random.seed(42) makes the sample reproducible. A dataset that changes every rebuild makes score changes uninterpretable — you cannot tell a model regression from a sampling difference.
• content_hash dedupes near-identical queries. Without it, fifty copies of "where is my order" (the most common phrase in any support log) can dominate a 120-case set and push pass rates up without measuring anything new.

The gold answer on each case is written once, by a person, at sampling time. That label is the contract every later scorer — human or LLM judge — is checked against.

A case without a label is an anecdote. What the label looks like depends on how many correct outputs exist:

The rubric row is the one teams skip. Asking a judge to "grade this answer" produces scores that drift with the judge's mood; asking it to score correctness, completeness, and safety against anchored examples produces scores that reproduce. The judge prompt itself is the subject of LLM-as-Judge Techniques.

Once labeled, the dataset gets partitioned. One set used for everything is a set you will overfit within a month:

raw production logs
        │
        ▼
┌──────────────────────────────┐
│  dedup + stratify + label    │
└──────────────────────────────┘
        │
        ├── regression (40%) ── run on every change; must not drop
        ├── exploratory (40%) ── iterate freely; may be messy
        └── holdout (20%)  ──── untouched until release decisions

The regression split is the gate. The exploratory split is where prompt edits are tuned. The holdout split is never tuned against — it exists so a release decision is made on cases the team has not already adapted to. Tuning against the holdout converts it into a second exploratory set, and the team loses its only honest estimate of production behavior.

A dataset is a living artifact, and the loop that keeps it alive is short: every production failure becomes a labeled case, permanently.

def promote_failure(failure: dict, path: str = "evals/support-v1.jsonl") -> None:
    case = {
        "input": failure["input"],
        "expected": failure["expected"],   # written by a person, once
        "added": "2026-09-30",
        "source": failure["ticket_id"],
    }
    with open(path, "a", encoding="utf-8") as fh:
        fh.write(json.dumps(case) + "\n")   # append-only, diffable

The source field is what makes the dataset auditable: any case can be traced back to the incident that created it, and any case can be defended in review by pointing at a real user interaction. Cases added between experiments shift the bar, so new failures are batched and added after each evaluation cycle, not mid-run.

Failure modes that rot a dataset:

• Sampling only easy traffic. Pass rates flatter the system; the long tail is where the tickets come from.
• Leaking the holdout into tuning. The release estimate becomes optimistic, and the surprise arrives after launch.
• One labeler, no rubric. Scores drift with whoever happens to judge; the label is the contract, so write the rubric before the labels.
• Never adding failures. The dataset measures last quarter's product while the product moves on.

The dataset you build here is the input to everything downstream: the offline harness runs it on every change, and the regression suite pins a slice of it against provider drift. A dataset that grows with the system is what makes those gates meaningful.