Evaluating Memory Quality

The complaint that starts every memory investigation is unanswerable on its own: "it forgot what I told it." There is no ground truth in production logs — you cannot score an answer when nobody recorded what the answer should have been. So the suite manufactures ground truth: write a fact into a session, bury it under real turns, then ask.

CASES = [
    {"seed": ["My standup moved to 9:15am UTC starting Monday."],
     "bury": 40,
     "probe": "What time is my standup?",
     "expect": "9:15"},
    {"seed": ["Decision: ship without the export button."],
     "bury": 60,
     "probe": "Are we shipping the export button?",
     "expect": "without the export button"},
]

def evaluate(cases, session, ask, bury=None):
    hits = 0
    for c in cases:
        depth = bury if bury is not None else c["bury"]
        session.reset()
        session.load(seeded_history(c["seed"]))       # plant the fact
        session.bury(filler_turns(depth))             # then hide it under real dialogue
        answer = ask(c["probe"], session.history())
        hits += c["expect"].lower() in answer.lower()
    return hits, len(cases)

Input: 12 cases, each planting one fact and burying it under 40–120 filler turns before probing.

Output: (7, 12) — 58% recall, and every one of the 5 misses came from a case with 60 or more turns between the seed and the probe.

Three properties make this a measurement rather than a demo. The expected value is known before the model answers, so scoring is mechanical. The filler is real turns — trimmed, summarized, re-ordered by whatever the session under test does — because synthetic filler never exercises the retention policy that usually causes the loss. And distance is a parameter: varying bury turns a single suite into a curve, which is where the interesting failure lives.

The seeds must survive whatever compression the session applies, so a suite that plants a fact and then loses it inside summarization is measuring Summarizing Long Conversations as much as the store itself. Designing the datasets properly — paraphrases, distractors, unseeded negatives — is Building Eval Datasets.

Recall alone scores only half the failure space. A system that always repeats the first thing it heard is perfect on recall and unusable in practice, while one that invents confident details never fails a "did it forget?" test either. Four metrics cover the space:

Staleness is scored against a corrected conversation — the fact exists twice, and only the newer value is true:

STALE_CASES = [
    {"seed":    ["Standup is at 10:00."],
     "correct": ["Actually, 14:00 works better for me."],
     "probe":   "What time is my standup?",
     "accept":  "14:00",
     "reject":  "10:00"},
]

def staleness(cases, session, ask):
    repeats = 0
    for c in cases:
        session.reset()
        session.load(seeded_history(c["seed"] + c["correct"]))
        answer = ask(c["probe"], session.history()).lower()
        if c["reject"] in answer and c["accept"] not in answer:
            repeats += 1                             # asserted the dead value
    return repeats, len(cases)

Input: 10 correction cases, each with the old value and the new value both present in the log.

Output: (2, 10) — 20% staleness: two answers insisted on the superseded time.

The false-memory half works the same way, pointed at negatives. Ask about something the log never contains — "what editor do I prefer?" when no editor was ever mentioned — and count any concrete answer as a failure while counting an abstention as a pass. Together the two directions separate the three real bugs: the memory dropped a fact (recall), the memory kept the wrong version (staleness), or the memory never had it and the model supplied one anyway (false recall). Each maps to a different fix in Managing Conversation History, Semantic Memory with Vectors, and the prompting layer respectively.

A single accuracy number hides the only fact worth knowing: memory systems do not degrade evenly. They hold flat and then fall off a cliff at the point where the retention policy starts discarding — so the suite runs at several session lengths and reports the curve.

LENGTHS = [10, 40, 100, 250]

def sweep(cases, session, ask):
    report = {}
    for n in LENGTHS:
        hits, total = evaluate(cases, session, ask, bury=n)   # same cases, re-buried deeper
        report[n] = round(hits / total, 3)
    return report

Input: the 12-case suite against a session using a 1,500-token window with a rolling summary, each case re-buried at every length.

Output: {10: 1.0, 40: 0.92, 100: 0.58, 250: 0.25} — flat, then a knee between 40 and 100 turns.

The knee is the deliverable. It tells you which mechanism is failing: recall collapsing at exactly the length where trimming begins indicts the retention policy; staleness rising while recall holds indicts the summary; both flat with false-memory climbing indicts the extraction gate in the store. Fixing any of them without the curve means you can only claim a new single number, not a repaired behavior.

Failure modes in the harness itself:

• Substring scoring — expecting 9:15 accepts 19:15, and expecting 10:00 inside "not 10:00" counts as a hit. Score on normalized answers with boundaries, and treat the rejected value as a failure even when the accepted one also appears.
• Identical filler every run — the suite then measures one lucky ordering. Shuffle filler and vary the seed phrasing.
• One seed wording per fact — models that pattern-match the planted sentence score well and still miss the fact when the user phrases it differently.
• Flaky gates — sampling at nonzero temperature makes the same commit pass or fail. Fix temperature or run each case three times and take the majority.
• Two changes per run — altering the trim policy and the model at once makes the curve unattributable.
• Aggregate-only reporting — one blended score lets a recall regression hide behind an abstention improvement. Publish all four metrics per length.

Wiring the sweep into CI so a merge that moves the knee blocks the build is Regression Suites; the harness structure it plugs into is Offline Eval Harness.