Assembling Context for the Model

The support bot returned: "The cap is 4000 EUR. The cap is 4000 EUR. The cap is 4000 EUR." Three sentences, one fact, and the passage that named the 90-day eligibility window never made it into the prompt at all.

Nothing was wrong with retrieval. Five passages came back ranked, and two of them were the same paragraph — one from the handbook, one from the FAQ page that quoted it — plus a third chunk that overlapped the first by 80% because the chunker uses a sliding window. Three of five slots went to one fact. The prompt had a budget in tokens, and duplication spent it twice.

Deduplication runs before anything else touches the list:

import re

def norm(text: str) -> set[str]:
    return set(re.sub(r"[^a-z0-9 ]", "", text.lower()).split())

def dedupe(passages: list[dict], threshold: float = 0.7):
    kept, kept_toks, dropped = [], [], 0
    for p in passages:                       # list arrives in rank order
        toks = norm(p["text"])
        if toks and any(len(toks & k) / len(toks | k) >= threshold for k in kept_toks):
            dropped += 1
            continue
        kept.append(p)
        kept_toks.append(toks)
    return kept, dropped

Input:

1 "The relocation cap for Berlin is 4000 EUR."          score 0.94
2 "Relocation cap for Berlin is 4000 EUR. Apply within 90 days of arrival."  score 0.91
3 "The relocation cap for Berlin is 4000 EUR."          score 0.90   (FAQ page)
4 "Parental leave is 14 weeks in Germany."              score 0.62

Output: kept 3 of 4, dropped 1

Two lines decide the behavior:

• The list is walked in rank order, and each passage is compared only against the ones already kept. The highest-ranked copy therefore wins, and a duplicate can never evict the passage that outranked it.
• The comparison is Jaccard overlap on normalized tokens. Exact-match dedupe would have missed rows 1 and 2 — they differ by a trailing sentence — while a threshold high enough to catch it would start merging genuinely similar policy pages.

Notice what dedupe does not do: it does not delete the overlap between two legitimately different chunks of the same document. Overlap is how a sliding-window chunker keeps a sentence from being cut in half; the assembler's job is to stop the same fact from occupying three slots, not to eliminate window overlap everywhere.

Deduplication returns budget, but budget only helps if the rest of the pipeline knows how to spend it. That is the next two steps: order, then trim.

Context assembly produces a block of text the model reads and answers from. Two properties of that block change the answer: which passages appear first, and whether a marker like [2] means anything after generation.

Order. Long contexts show primacy effects: instructions and evidence near the top get more weight than equivalent material buried at position 3,000. Retrieval order is a similarity ranking, not a reading order, so the assembler decides explicitly:

For a mixed corpus, take one passage per source per round, then fall back:

def diverse_order(passages: list[dict]) -> list[dict]:
    pools: dict[str, list[dict]] = {}
    for p in sorted(passages, key=lambda p: p["score"], reverse=True):
        pools.setdefault(p["doc"], []).append(p)
    ordered = []
    while pools:
        for doc in list(pools):
            ordered.append(pools[doc].pop(0))   # one per source per round
            if not pools[doc]:
                del pools[doc]
    return ordered

Input: passages from handbook (0.94, 0.71), faq (0.91), wiki (0.62). Output order: handbook(0.94), faq(0.91), handbook(0.71), wiki(0.62) — the strong second passage from handbook waits one round instead of stacking on top.

Number. The number the model writes in its answer is only meaningful if you control the mapping:

def number(passages: list[dict]) -> str:
    lines = []
    for i, p in enumerate(passages, 1):        # 1-based, dense, no gaps
        lines.append(f"[{i}] ({p['section']}) {p['text']}")
    return "\n\n".join(lines)

The prompt then says: cite the passage number after each factual sentence, using [n]. Because numbering happens after dedupe and ordering, [2] refers to the second line the model actually saw — not to the second result the vector database returned. Evict a passage during trimming and the surviving numbers are re-issued densely, so no citation can point at a gap. Whether the model's markers are then validated is the job of citation grounding; assembly's duty is to keep the mapping honest.

Retrieval will happily return more passages than fit. Left alone, the request either overflows the window — a 4000 context_length_exceeded in production — or the service silently truncates from the end, which throws away passages nobody ranked.

The budget is arithmetic, and the answer gets paid first:

def count_tokens(text: str) -> int:
    return max(1, len(text) // 4)              # ~4 chars/token estimate; swap in the real tokenizer

def assemble(passages: list[dict], question: str, window=8192, max_output=1024, margin=128):
    budget = window - max_output - margin       # the answer is reserved before the prompt is built
    left = budget - count_tokens(question) - 60 # 60: instruction boilerplate + [n] markers

    kept = []
    for p in passages:                         # already deduped and ordered best-first
        cost = count_tokens(p["text"]) + 4
        if cost > left:
            break                              # evict this one and everything below it
        kept.append(p)
        left -= cost

    context = "\n\n".join(f"[{i}] {p['text']}" for i, p in enumerate(kept, 1))
    return context, budget - left

Input: 5 passages (820, 610, 700, 480, 390 tokens), question of 40 tokens, defaults above. Output:

kept 3 of 5 passages, 2130 / 7040 prompt tokens used

Three decisions:

• max_output and margin come out first. Reserving the answer after filling the prompt means the generation gets truncated mid-sentence, which users read as a broken product.
• The loop breaks instead of skipping. Continuing past an ill-fitting passage to admit a smaller, lower-ranked one packs in more text but makes the order lie: the model sees evidence it was told ranks fifth sitting above evidence that ranks third. Rank fidelity beats density here.
• Reaching inside kept and renumbering from 1 keeps the citation contract intact after eviction.

Everything above is passage-level budgeting. The rest of the request — system prompt, conversation history, the reserved output itself — is the larger arithmetic covered in context window budgeting, and the assembler sits inside that calculation as one line item.