Citations and Grounded Answers

A finance assistant wrote: "Reimbursements settle in 14 days [2]." Passage [2] in the prompt said 30 days. The marker was well-formed, numbered correctly, and attached to a sentence that contradicted the passage it pointed at.

Three defects were hiding in that one line, and only the third was a model problem. First, nothing in the prompt said when a marker is required, so the model used them decoratively. Second, nobody checked afterward whether the ids it wrote exist in the set that was sent. Third, even a valid id was never compared back to the text it claims to summarize.

The contract has to be stated, then enforced:

You answer ONLY from the numbered passages below.

Rules:
1. After every factual sentence you take from a passage, append its number in
   brackets, like this: Employees are reimbursed within 30 days [2].
2. Never cite a number you were not given.
3. If the passages do not contain the answer, reply exactly:
   "Not covered by the provided documents."
4. Do not merge two passages into one sentence unless both numbers appear.

Applied to "How long does reimbursement take?" with the passage set from context assembly:

Output:

Reimbursements settle within 30 days of approval [2]. Claims for per diem are
paid on the same schedule [2].

Asked a question the passages do not cover — "does the plan cover home office furniture?" — the same contract produces:

Not covered by the provided documents.

Rule 1 converts citation from a stylistic choice into a per-sentence obligation, which is what makes mechanical verification possible later: a marker is now expected data, not decoration. Rule 4 closes the loophole where a model blends two passages and cites whichever number it remembers. Rule 3 matters most — an instruction to cite cannot stop a fabrication when the evidence simply is not there; only an explicit escape hatch does.

A contract only binds a cooperative model. Everything after this chapter assumes the answer may violate it, and exists to catch the violations.

The retrieved set is known server-side: you chose the passages and numbered them. So validating markers is set arithmetic, and it costs microseconds.

import re

SENTENCE = re.compile(r"(?<=[.!?]) +")           # lookbehind keeps the punctuation
MARKER = re.compile(r"\[(\d+)\]")

def check(answer: str, passages: list[str]) -> dict:
    valid = set(range(1, len(passages) + 1))
    cited = {int(m) for m in MARKER.findall(answer)}
    sentences = [s for s in SENTENCE.split(answer) if s.strip()]
    # a sentence stating a quantity with no marker is an unattributed claim
    unmarked = [s for s in sentences
                if re.search(r"\d", s) and not MARKER.search(s)]
    return {
        "cited": sorted(cited),
        "bad_ids": sorted(cited - valid),         # ids we never sent
        "unmarked": unmarked,
    }

Input: the answer above with passages of length 3. Output:

{'cited': [2], 'bad_ids': [], 'unmarked': []}

Input: an answer containing "Refunds take 14 days [7]." and "Reimbursements settle in 14 days." Output:

{'cited': [7], 'bad_ids': [7], 'unmarked': ['Reimbursements settle in 14 days.']}

Two different failures surfaced, and they are handled differently:

The distinction is worth defending in an interview: set arithmetic catches references that cannot resolve; it says nothing about whether the cited sentence is supported. Full support checking is a separate, slower evaluation — an entailment check or an LLM-as-judge pass — and belongs offline in sampling, not in the request path.

What must never happen is serving an unverifiable marker as though it were checked. Strip it, or drop the sentence with it.

A citation is only useful if the reader can act on it. That requires two more pieces: an answer that admits when evidence is missing, and a payload that turns [2] into something clickable.

The refusal rule from chapter one does the first job. Make it checkable by matching an exact string rather than asking the model to "be honest":

REFUSAL = "Not covered by the provided documents."

def resolve(answer: str, passages: list[dict]) -> dict:
    markers = [int(m) for m in re.findall(r"\[(\d+)\]", answer)]
    if answer.strip() == REFUSAL:
        return {"status": "no_evidence", "sources": []}

    sources, seen = [], set()
    for i in sorted(set(markers)):
        p = passages[i - 1]                       # numbering is 1-based and dense
        if p["source_url"] not in seen:           # one chip per document, not per chunk
            seen.add(p["source_url"])
            sources.append({
                "label": p["section"],
                "url": p["source_url"],
                "quote": p["text"][:160],
            })

    bad = [i for i in markers if not 1 <= i <= len(passages)]
    return {"status": "ok" if not bad else "unverified",
            "sources": sources}

Input: the answer [2] with three passages whose URLs are hr.example/policy and hr.example/faq. Output:

{'status': 'ok',
 'sources': [{'label': 'Reimbursement', 'url': 'https://hr.example/policy',
              'quote': 'Reimbursements settle within 30 days of approval...'}]}

Three choices in that function:

• passages[i - 1] converts the model's [n] back to the row you sent. It is safe only because assembly numbers densely after trimming — renumbering before trimming would silently bind [2] to the wrong passage, which is precisely the class of bug this pipeline keeps guarding against.
• Dedupe by source_url keeps the UI honest: five chunks from one document are one source the reader can open, not five chips.
• status: "unverified" rather than dropping the answer on the floor. Unresolvable markers are rare and diagnosable; the incident log in RAG failure modes is where they get root-caused.

Users who see the section label and can open the document verify claims themselves — which is the actual purpose of the exercise. A citation the reader cannot follow is a citation-shaped decoration.