Iterating on a RAG System

Two engineers shipped improvements in the same week: one re-chunked the handbook, one rewrote the retrieval prompt. Reported quality dropped three points. Both changes stayed, each blamed the other, and the team could not settle it — because neither had a recorded baseline to compare against, and the "improvements" had been judged on whatever examples each person happened to try.

The fix costs an afternoon and prevents the whole category: a frozen set of questions with reference answers, written before anyone touches a component.

import json
from collections import Counter

def load_golden(path: str) -> list[dict]:
    cases, seen = [], set()
    with open(path, encoding="utf-8") as fh:
        for line in fh:
            c = json.loads(line)
            assert c["id"] not in seen, f"duplicate id: {c['id']}"
            assert c["reference"].strip(), f"{c['id']} has an empty reference answer"
            assert c["gold_pids"], f"{c['id']} has no reference passages"
            seen.add(c["id"])
            c.setdefault("slice", "general")   # every case is attributable to a category
            cases.append(c)
    return cases

Input (golden.jsonl, one object per line):

{"id": "q-041", "question": "What is the relocation cap for Berlin?",
 "reference": "4000 EUR, claimed within 90 days of arrival",
 "gold_pids": ["handbook:9a17cc33:2"], "slice": "policy", "weight": 3}
{"id": "q-042", "question": "How long does reimbursement take?",
 "reference": "30 days from approval", "gold_pids": ["faq:5c04d7e1:0"],
 "slice": "billing"}

Output:

loaded 126 cases | slices: billing 34, policy 41, benefits 51 | weight 3 on 18

Each assertion defends a property that quietly rots: duplicate ids make two results indistinguishable in a diff; an empty reference means a case that can never be scored; missing gold_pids makes retrieval recall unmeasurable for that question. And weight plus slice exist so "quality went up" can be decomposed — a gain that comes from easy general questions is not a gain.

Three rules for keeping it honest:

1. Reference answers are written once, by a person, and then edited only through a review. A reference regenerated by the same model you are evaluating grades itself.
2. The set is frozen during an experiment. Adding cases mid-flight changes the denominator and invalidates every delta recorded so far.
3. New production failures get appended afterward as permanent cases, so the set grows into a regression corpus rather than a snapshot of day one.

Which components are fair game for an experiment, and in what order, is decided next — one at a time or not at all.

A bundled change cannot be attributed. Re-chunk with a new size and swap the embedding model and reword the retrieval instruction, and any delta belongs to the bundle. So each experiment varies exactly one component and reports two numbers that are never blended into a single "score":

import re

def recall_at_k(hits: list[dict], gold: set[str]) -> float:
    return 1.0 if any(h["pid"] in gold for h in hits) else 0.0

NUM = re.compile(r"\d+(?:[.,]\d+)?")

def faithfulness(answer: str, prompt: str) -> float:
    claims = NUM.findall(answer)                  # quantities are the claims that hurt
    if not claims:
        return 1.0                                # nothing checkable, nothing to penalize
    prompt_nums = set(NUM.findall(prompt))
    return sum(1 for c in claims if c in prompt_nums) / len(claims)

def evaluate(cases, retrieve, generate, k=5):
    rec, faith = 0.0, 0.0
    for c in cases:
        hits = retrieve(c["question"], k)
        rec += recall_at_k(hits, set(c["gold_pids"]))
        prompt = "\n".join(h["text"] for h in hits)
        faith += faithfulness(generate(c["question"], prompt), prompt)
    n = len(cases)
    return {"recall@k": round(rec / n, 3), "faithfulness": round(faith / n, 3)}

Input: 126 golden cases against a baseline, then a single re-chunk. Output:

                     recall@k   faithfulness   Δrecall   Δfaith
baseline                0.780        0.910         —         —
chunk-600-no-overlap    0.706        0.921      -0.074    +0.011

The two columns explain each other. Bigger chunks covered the handbook's long tables better — faithfulness ticked up because the numbers landed in the prompt — while recall fell because fewer distinct passages fit in the top-k. A blended score would have called this a wash; the split says retrieval got worse and generation got more support, which suggests keeping the new chunker only if retrieval is fixed separately (smaller k, a reranker, query rewriting).

faithfulness() here is deliberately a cheap proxy: it checks that quantities in the answer appear in the prompt, not that every sentence is entailed by it. It is deterministic, runs offline, needs no API calls, and catches the worst class of defect — invented numbers. When a proxy stops being enough, an LLM-as-judge fills the same slot without changing the harness.

Keep the experiment itself on one line: id, date, the component touched, the hypothesis, the two deltas, the worst slice delta, and the decision. An experiment log with a decision column is what stops the argument in the first chapter from recurring.

Aggregate movement is necessary but not sufficient. A change can lift overall recall while destroying the questions that actually lose money, and averages will hide it unless slices are compared explicitly.

def promote(baseline: dict, candidate: dict, protected=("billing", "policy"),
            min_gain=0.01) -> str:
    reasons = []
    if candidate["recall"] < baseline["recall"] + min_gain:
        reasons.append(f"recall gain under {min_gain}")
    if candidate["faithfulness"] < baseline["faithfulness"]:
        reasons.append("faithfulness dropped")
    harmed = [s for s in protected
              if candidate["slices"].get(s, 1.0) < baseline["slices"].get(s, 1.0)]
    if harmed:
        reasons.append(f"protected slice regressed: {harmed}")
    return "PROMOTE" if not reasons else "keep baseline — " + "; ".join(reasons)

Input: baseline recall 0.780, faithfulness 0.910, slices {billing 0.72, policy 0.81}; candidate recall 0.794, faithfulness 0.911, slices {billing 0.65, policy 0.83}. Output:

keep baseline — protected slice regressed: ['billing']

The candidate passed on aggregate — recall up 0.014, faithfulness non-decreasing — and still lost, because billing questions fell seven points. That is the whole argument for gates: a regression on a protected slice is a product incident waiting to be reported, and a merge that causes it should fail review before it reaches production.

The loop, complete:

freeze golden set --> change ONE component --> run eval --> record deltas
       ^                                                     |
       |                    recall down, faithfulness down    |
       |                 +------------------------------------+
       |                 v
       |            revert or fix, log the result anyway
       |
       +--- gains on aggregate, no protected slice harmed --> PROMOTE
                                                              (baseline := candidate)

Two properties make the loop survive contact with a real team:

• Rejects are logged with their numbers. "We tried 600-token chunks and it cost 7 points of recall" is knowledge; a reverted commit is not.
• The golden set changes only between experiments, when a production failure is promoted into a new case. Inside an experiment it is read-only.

Once promotion is mechanical, the pipeline components become interchangeable again: swapping the vector store for pgvector, adding a reranker, or re-chunking the corpus are all the same operation — one change, two numbers, two gates — and the harness built here is what tells you whether they helped.