Diagnosing RAG Failure Modes

An incident arrives as: "The bot told me parental leave is 20 weeks; the handbook says 14." The instinct — shared by almost every new team — is to open the prompt and start rewording it. That instinct is wrong often enough to be expensive, because a generation-side edit cannot fix evidence that never reached the model.

The first question is not "what did the model say?" but "what did the model see?" Log the three ids together and the argument becomes mechanical:

answer supported by the prompt?
  |
  +-- gold passage in the prompt? --+-- no --> in the retrieved set?
  |                                  |           |-- no  --> RETRIEVAL
  |                                  |           |        (chunking, embeddings,
  |                                  |           |         query rewriting)
  |                                  |           +-- yes --> ASSEMBLY
  |                                  |                    (evicted by dedupe or budget)
  |                                  |
  |                                  +-- yes --> GENERATION
  |                                               (instruction, model, decoding)
  |
  +-- yes --> SOURCE or LABEL
              (gold annotation wrong, document stale)

Read it as a series of eliminations. Retrieval is guilty when the evidence never entered the top-k. Assembly is guilty when retrieval found it and the token budget threw it away — a distinct bug with a distinct fix, and invisible unless you log prompt_pids separately from retrieved. Generation is guilty only once the right passage is provably sitting in the prompt.

The traversal itself is code, and it is the first thing to run on any report:

import re
from collections import Counter

REFUSAL = "Not covered by the provided documents."

def attribute(log: dict) -> str:
    gold = set(log.get("gold_pids", []))
    retrieved = {p["pid"] for p in log["retrieved"]}
    prompt = set(log["prompt_pids"])
    answer = log["answer"].strip()

    if not gold:
        return "eval-gap: no reference passage logged"
    if not (gold & retrieved):
        return "retrieval: gold passage outside top-k"
    if not (gold & prompt):
        return "assembly: retrieved then evicted before generation"
    if answer == REFUSAL:
        return "generation: refused despite evidence in the prompt"

    cited = {int(m) for m in re.findall(r"\[(\d+)\]", answer)}
    if cited - set(range(1, len(prompt) + 1)):
        return "generation: citation id not in the prompt"
    if re.search(r"\d", answer) and not cited:
        return "generation: numeric claim with no marker"
    return "source-or-label: prompt supported the answer"

Input: a log whose gold_pids = ["handbook:9a17:2"], retrieved contains it, prompt_pids does not. Output:

'assembly: retrieved then evicted before generation'

Every branch names a different owner. Fixing an assembly eviction by rewriting the prompt produces a second incident a week later, and the log would have said so.

These are the entries that recur across RAG systems. Severity is not how likely the failure is — it is what happens to the user when it fires:

• critical — a confidently wrong answer is served, or a citation is fabricated
• high — a common path returns wrong or missing information
• medium — quality degrades but the answer is still usable

Two patterns are worth memorizing because they invert the obvious fix:

High recall, low faithfulness means retrieval is fine and generation is drifting — a prompt, model, or temperature problem. Re-chunking makes it worse by changing the one half that already works.

Low recall on one slice only — the benefits questions, the German pages — usually means vocabulary mismatch or a parse defect on that document type, not a global embedding problem. The slice tells you where to look before you change anything global.

Diagnosis that stops at "it's the model" is a coin flip. Diagnosis that names a stage and a signal is an actionable ticket, and the difference is which log fields you captured.

A report you cannot replay is a report you can only fix once. Store the retrieved set with the answer at request time, and every incident becomes a reproducible input to attribute() long after the request finished.

def replay(logs: list[dict]) -> dict:
    counts = Counter(attribute(log) for log in logs)
    return dict(sorted(counts.items(), key=lambda kv: -kv[1]))

Input: four stored logs — one with gold outside top-k, one retrieved-then-evicted, one refusal despite evidence, one clean. Output:

{'retrieval: gold passage outside top-k': 1,
 'assembly: retrieved then evicted before generation': 1,
 'generation: refused despite evidence in the prompt': 1,
 'source-or-label: prompt supported the answer': 1}

Replaying a batch rather than a single case changes the decision you make. One retrieval failure might be a fluke; four incidents bucketing into three different stages says your problems are not concentrated, and a single global knob change will not fix them.

The same replay is also where fixes get validated. After re-chunking or raising the budget, re-run attribute() over the stored logs and compare bucket counts — a fix that moves incidents between buckets rather than reducing them is a fix that renamed the problem. That comparison loop, with a frozen question set and recorded metric deltas per change, is what the RAG iteration workflow topic formalizes.

Minimum fields to log per request: query, retrieved (pid, score, section), prompt_pids, answer, model, prompt_version, gold_pids where an eval label exists, and a timestamp. Missing prompt_pids is the single most common gap — without it, assembly and retrieval failures are indistinguishable and every diagnosis collapses into "retrieval was bad."