Semantic Memory with Vectors

Support escalated a chat where the assistant opened with "Welcome back, Dr. Alvarez" for a user who had never given a name — and, a week later, a second user started receiving the same greeting. The name had been extracted once from an unrelated conversation, embedded without provenance, and matched to everyone whose recent turns were loosely adjacent.

Durable memory is a write path with four stages: extract, embed, upsert with source metadata, gate. Writing is the easy half; provenance is what makes a belief debuggable six months later.

EXTRACT = """Extract durable facts about the user from this turn.
Return at most 2. Skip pleasantries, questions, and anything about
the assistant. Each fact is one standalone third-person sentence."""

def remember(pool, user_id, conv_id, turn, llm, embed):
    facts = llm.extract(EXTRACT, turn.content)        # candidates, not memory yet
    facts = [f for f in facts
             if cosine(embed(f.text), embed(turn.content)) >= 0.62]
    if not facts:
        return 0                                      # small talk writes nothing

    vecs = embed([f.text for f in facts])             # one batched call, not one per fact
    with pool.cursor() as cur:
        for fact, vec in zip(facts, vecs):
            cur.execute(
                """INSERT INTO memories
                       (user_id, text, embedding, source_conv, source_turn,
                        content_hash, created_at, last_seen)
                   VALUES (%s, %s, %s::vector, %s, %s, %s, now(), now())
                   ON CONFLICT (user_id, content_hash)
                   DO UPDATE SET embedding = EXCLUDED.embedding,
                                 last_seen = now()""",
                (user_id, fact.text, str(vec), conv_id, turn.seq, digest(fact.text)),
            )
    return len(facts)

Input: the turn "Call me Priya, and I'm always in UTC."

Output: 1 — one row: u-17 | Priya prefers UTC. | conv c-9f2 | turn 4

The pre-write similarity check compares the extracted fact against the turn it came from. An extractor that invents a credential the user never stated scores low against its own source and is discarded before storage — the first defense against a false memory, and the cheapest one. content_hash, a digest of normalized text, turns a restated fact into an update rather than a duplicate, so a preference repeated a year later refreshes last_seen instead of adding a twin. source_conv and source_turn are the provenance that turns "why is it saying that?" into one SQL query.

Where the memories table itself lives is Choosing a Vector Store; the distance operator and index syntax are Vector Search with pgvector.

Every memory read asks the same question: given what the user just said, which stored facts belong in this request? The probe is the current turn — not the whole conversation, not the last summary — because the current turn is what the answer must be conditioned on.

def recall(pool, user_id, probe, embed, k=5, min_sim=0.78):
    vec = str(embed([probe])[0])
    with pool.cursor() as cur:
        cur.execute(
            """SELECT text, 1 - (embedding <=> %s::vector) AS sim, source_conv
               FROM memories
               WHERE user_id = %s AND deleted_at IS NULL
               ORDER BY embedding <=> %s::vector
               LIMIT %s""",
            (vec, user_id, vec, k),
        )
        rows = cur.fetchall()
    # distance becomes cosine similarity; a low score is coincidence, not memory
    return [r for r in rows if r.sim >= min_sim]

Input: probe "What time did I say my standup moves to?" against a store holding Standup moves to 9:15 UTC (0.86), Uses a MacBook Pro (0.71), Dislikes emoji in emails (0.64).

Output: [('Standup moves to 9:15 UTC', 0.86, 'c-9f2')] — one row; the other two fall under min_sim.

Four decisions carry the read. ORDER BY embedding <=> ... lets the vector index do the sorting, so the database returns k candidates at index speed rather than shipping the whole table to Python. The min_sim floor is applied after the limit, and it is not optional: top-k with no floor returns k rows no matter how irrelevant, which is precisely how an unrelated name reached users who never said it. Filtering deleted_at IS NULL makes a user's delete take effect on the very next request. And the user_id predicate is inside the statement again, for the same reason it is on turns.

Retrieved memories enter the request as a system note beside the rolling summary from Summarizing Long Conversations, and what the query embedding may and may not discriminate is Cosine Similarity.

Users do not experience memory as one mechanism. They experience three, with different lifetimes, different costs, and different ways of being wrong:

             per request               per session                  durable
       ┌──────────────────────┐  ┌───────────────────────┐  ┌─────────────────────────┐
       │ context window       │  │ conversation log       │  │ semantic memory store   │
       │ system prompt        │  │ all turns by seq       │  │ embedded, deduplicated  │
       │ recent turns         │──▶│ trimmed by policy      │──▶│ survives sign-out       │
       │ recalled memories    │  │ + head summary         │  │ user-visible, deletable │
       └──────────────────────┘  └───────────────────────┘  └─────────────────────────┘
          volatile, cheap           complete, session-bound       expensive, curated
          dies with the request     dies with retention window    must be correctable

Only the rightmost tier can stay wrong for months, so only it needs a shelf the user can see:

@app.get("/memories")
def list_memories(dep=Current_user):
    return db.select(
        "SELECT text, source_conv, created_at FROM memories "
        "WHERE user_id = %s AND deleted_at IS NULL", [dep.id])

@app.delete("/memories/{memory_id}")
def forget(memory_id, dep=Current_user):
    # ownership lives in the statement, the same rule as turn reads
    db.execute("UPDATE memories SET deleted_at = now() "
               "WHERE id = %s AND user_id = %s", [memory_id, dep.id])

Soft delete rather than a hard DELETE: incident review still needs to know what the system believed, while the read path — which already filters deleted_at IS NULL — hides it immediately.

Failure modes to design against:

• Write-through without a gate — every greeting lands in the store and retrieval returns noise. Threshold the write, not only the read.
• Probing with the whole conversation — embedding 40 turns averages away the actual question, so retrieval matches the topic instead of the need.
• No floor under top-k — LIMIT 5 returns five rows always; without min_sim, unrelated facts are injected with full confidence.
• Silent persistence — a memory the user cannot inspect is one they cannot correct. The wrong-name incident shipped without a delete endpoint.
• Missing tenant predicate — a shared store without WHERE user_id is the same hole as an unscoped turn read in Managing Conversation History.

Whether the store actually remembers — and whether it remembers things that were never said — is measured in Evaluating Memory Quality.