Strategies for Long Inputs

A signed vendor contract runs to 312,000 tokens. The model's usable window holds far less, and even a model that claims the full length answers better when the relevant clause is not buried under 300,000 neighbors. The real question is not how to fit the document — it is which parts of it the answer actually needs.

Four strategies, each discarding something different:

The choice follows the question shape:

broad question, whole document matters   -> map-reduce
narrow question, answer is a passage     -> retrieval (top-k chunks)
position is meaningful (tail = newest)   -> truncation
input is short anyway                    -> single shot on one model

Long-context models move the boundary without changing the principle: answers degrade when relevance is diluted across a huge input, so stuffing the full document still loses to sending a focused one. Map-reduce is the coverage strategy and gets the worked example next; truncation and retrieval each have their own chapter. The cutting itself starts from chunk definitions — Fixed-Size Chunking and Semantic Chunking — and every byte you keep is priced by Estimating Token Costs.

Map-reduce summarization reads every chunk, summarizes each independently, then merges the summaries until one fits. The call count — not the algorithm — is what you pay for, so packing matters:

def chunk(tokens, size):
    return [tokens[i:i + size] for i in range(0, len(tokens), size)]

def map_reduce(tokens, size, summarize, condense):
    partials = [summarize(p) for p in chunk(tokens, size)]   # map: each chunk alone
    calls = len(partials)
    while len(partials) > 1:
        packed, batch, cost = [], [], 0
        for p in partials:
            if cost + len(p) > size:                        # pack partials until full
                packed.append(condense(batch))
                batch, cost = [], 0
            batch.append(p)
            cost += len(p)
        if batch:
            packed.append(condense(batch))
        calls += len(packed)
        partials = packed
    return partials[0], calls

Input: 312,000 tokens, size=6,000, with summarize and condense each returning 400-token partials.

Output: (400-token final summary, 57 calls)

Walk the arithmetic: 312,000 / 6,000 = 52 map calls producing 52 x 400 = 20,800 tokens of partials; the first reduce pass packs 15 partials per call (15 x 400 = 6,000 exactly), so 20,800 needs 4 calls yielding 1,600 tokens; one final call merges those into the 400-token answer. 52 + 4 + 1 = 57 calls, reading 312,000 + 20,800 + 1,600 = 334,400 input tokens — only about 7% overhead above the document, because each higher level re-reads summaries instead of source text.

Two lines deserve the explanation. cost + len(p) > size is the same budget arithmetic as Context Window Budgeting, applied to merge batches rather than chat turns. while len(partials) > 1 makes the reduce recursive until one summary remains — that loop is where a runaway can hide if condense fails to shrink its input.

What map-reduce loses: exact quotes and one-off numbers. A figure mentioned once in chunk 37 may not survive two rounds of summarization, so arithmetic questions over the document should count on retrieval instead.

Truncation is the cheapest strategy — one call, no search infrastructure — and the only one whose losses are completely silent.

def keep_recent(turn_sizes, limit):
    kept, used = [], 0
    for size in reversed(turn_sizes):
        if used + size > limit:
            break                      # everything older stays out, with no error
        kept.append(size)
        used += size
    return list(reversed(kept)), used

def keep_ends(text, head, tail):
    if len(text) <= head + tail:
        return text
    return text[:head] + f'\n[...{len(text) - head - tail} chars elided...]\n' + text[-tail:]

Input: keep_recent([1240, 1180, 1305, 1150, 1260], 4000)

Output: ([1305, 1150, 1260], 3715) — the two oldest turns (1,240 and 1,180 tokens) are gone.

The failure is what "gone" means. If turn one carried the user's original request and turn two carried the clarifying answer, the model now faces a conversation whose question it cannot see. Nothing errors; the response just drifts. keep_ends has the same shape for documents: policy texts and reports put the ask at the top and the conclusion at the bottom, and the elided middle is exactly where the clause you needed lived.

Failure modes to name before shipping a truncation policy:

• Silent question loss — history truncated past the turn that stated the task.
• Mid-sentence cuts — the model receives half a table or a paragraph ending in a conjunction.
• Recency assumed as relevance — true for chat tails, false for documents where facts sit anywhere.
• Elision markers misread — a literal [...elided...] can be quoted by the model as document content.

When facts can sit anywhere in the input, prefer map-reduce for coverage or retrieval for narrow queries. History-specific trimming lives in Conversation History, and the eviction rules that decide which turns survive a single request are in Context Window Budgeting.

Retrieval inverts the default: decide in advance which few passages deserve the budget, then let everything else stay out of the prompt. Chunk the document (Fixed-Size Chunking), embed the query, take the top-k matches — and the prompt carries roughly 12 chunks x 400 tokens = 4,800 tokens instead of 312,000, a 65x cut before pricing.

The trade is structural. Retrieval answers narrow questions well and misses anything the query wording does not surface: a fact phrased differently from the question ranks low or not at all. Search quality becomes the ceiling on answer quality, and no prompt edit rescues wrong chunks — Hybrid Search Basics covers widening recall, and RAG Failure Modes catalogues what it looks like when retrieval is the broken layer.

This is also why a larger window does not retire the strategy. Relevance dilutes: the more irrelevant text surrounds the one clause you need, the more often it gets missed or blended with neighboring passages — and the input bill grows linearly while answer quality does not. Long-context models reduce how often you must chunk, but they do not remove the decision about what deserves attention. The RAG Pipeline Overview builds the retrieval path; this topic's job is knowing when to reach for it.

Recap the decision from the first chapter with all four on the table: broad coverage → map-reduce; narrow, foreseeable questions → retrieval; meaningful position → truncation; short input → single shot. Whatever gets assembled, the instructions for how to use it belong in System Prompt Design.