Summarizing Long Conversations

Compression is the moment a chat system agrees to forget something. Before the threshold, every turn is kept verbatim and the model sees exactly what was said. After it, the head of the conversation is gone and one paragraph stands in for it. The question is not whether to summarize — the window forces that — it is what you have promised to lose, and when.

Two parameters hold the policy:

THRESHOLD = 6000   # fire only when history exceeds this many tokens
KEEP      = 1600   # raw dialogue kept verbatim, newest first

def compress(history, llm):
    total = sum(t.tokens for t in history)
    if total <= THRESHOLD:
        return history                      # under budget: no call, no cost, no loss

    system, turns = history[0], history[1:]
    head, tail = split_by_tokens(turns, total - system.tokens - KEEP)
    note = Turn("system", "Conversation so far:\n" + llm.summarize(head))
    return [system, note] + tail            # head is replaced, not appended beside

Input: 18 turns totalling 7,420 tokens (96 of them the system prompt); the newest 1,600 body tokens are the tail, 6 turns.

Output: 8 entries — system (96), summary note (254), 6 recent turns (1,600) — 1,950 tokens instead of 7,420.

Three lines carry three decisions. The total <= THRESHOLD guard makes compression on-demand rather than per-turn: a summarization call costs money, adds a round trip of latency, and destroys detail, so it should fire roughly once per session instead of once per message. split_by_tokens(..., total - system.tokens - KEEP) defines the sacrifice as a token count, not a turn count — a user who pastes a 900-token document pulls the compression point forward, and a "drop the first 5 turns" rule would not notice. [system, note] + tail is a replacement: appending the note while keeping the head grows history faster than before it.

Compress after each write, before the next request assembles. What gets sent from there is the trimming decision in Managing Conversation History, and the arithmetic that fixes THRESHOLD and KEEP lives in Context Window Budgeting.

A summary is lossy compression, and lossy in the wrong places is worse than none at all. "They discussed deployment plans" is a valid summary that also deletes the one fact the next turn needs. The contract has to separate what may be paraphrased from what must be copied character for character:

SUMMARY_PROMPT = """Summarize this conversation for use in later turns.

Copy VERBATIM — do not paraphrase, round, or reword:
  - decisions the user made ("ship without the export button")
  - open questions still unresolved
  - preferences and constraints (timezones, formats, limits, names)

Compress: greetings, restatements, attempts that failed.

At most 120 words. No preamble, no "The user..." prefix.

Conversation:
{transcript}
"""

Good output:

Decided: ship without the export button (turn 6). Open: whether billing
migrates before Friday. Prefers UTC and 24h time; no emoji in customer
email. Name: Priya Raman.

Bad output:

The user discussed a launch, raised concerns about billing, and shared
some preferences about how they like things handled.

The bad one passes a human glance and fails at turn 20, when the model needs 10:15 UTC and receives a mood. Three instructions make the difference: the verb copy rather than include, concrete categories with an example each, and the 120-word ceiling. Drop the ceiling and the verbatim list expands until the summary costs nearly as much as the turns it replaced.

Placement matters as much as wording. The note goes in as a system turn ahead of the surviving tail — [note] + recent_turns — not as a user message. A system entry is standing context: the model treats it as background it may rely on. Spoken as a user message it reads as a fresh claim to verify, and models will occasionally argue with it or attribute it to the wrong speaker. That is the same lever as persona and format rules in System Prompt Design.

Summarization is the one step in the pipeline that destroys data, which is exactly why the raw log must not participate in the destruction. Keep turns append-only and treat every summary as a derived view over them:

CREATE TABLE summaries (
  user_id         text   NOT NULL,
  conversation_id text   NOT NULL,
  version         int    NOT NULL,   -- 1, 2, 3 ... one active per conversation
  covers_from     int    NOT NULL,   -- seq of the first turn compressed
  covers_to       int    NOT NULL,
  text            text   NOT NULL,
  prompt_rev      text   NOT NULL,   -- which SUMMARY_PROMPT produced it
  created_at      timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (user_id, conversation_id, version)
);

Rollback is then a pointer move rather than a restore: mark version N inactive and rebuild the send path from version N-1, or from raw turns if none exists. Compression decides what is sent, never what is stored — the distinction is what makes an irreversible step reversible.

Failure modes worth naming:

• Chained compression — summarizing the previous summary compounds loss until only abstractions survive. Always compress from the raw head, never from the last note.
• Double compression — two workers cross the threshold concurrently and both write notes, so the model reads the same history twice. Guard the write with a version compare or a Postgres advisory lock.
• Mid-task compression — firing the threshold inside a tool loop can drop the tool result the next step depends on. Suppress compression between a tool call and its follow-up turn.
• Prompt revision without re-run — editing SUMMARY_PROMPT changes nothing for sessions already compressed; record prompt_rev and re-compress on demand.
• Silent rounding — the summarizer turns 9:15 into "morning" and a specific constraint into a vague one. That drift is measurable, and Evaluating Memory Quality is where it gets scored rather than noticed by users.

Summaries are the compact half of the design; the facts worth keeping forever, independent of any single session, are the subject of Semantic Memory with Vectors.