Query Rewriting and Expansion

The second turn of a conversation was "and what about the second one?" Retrieval embedded that string and returned passages about second — ordinals, rankings, second-quarter revenue. The referent was three turns back in the chat, and the chat is not in the index.

Two different defects produce that miss, and one stage fixes both:

• Reference without resolution. "the second one", "that plan", "the same error" carry their meaning in history the index has never seen.
• Under-specification. "the login thing again" — the user knows which system they mean; the query text does not say it.

The rewrite stage sits between the conversation and the retriever:

[history turns] + [latest turn]
        |
        v
   [ rewrite ] --- timeout / bad JSON / empty ---> latest turn used raw
        |                                              |
        v                                              v
 {query, facets[]} ---> [ retrieve per facet ] --> [ fuse ] --> [ rerank ]

Because the failure path rejoins the normal path, a rewrite outage degrades search to its current behavior instead of blocking it — a design constraint, not an optimization.

A rewrite is a retrieval-shaped transformation: it keeps the intent, resolves the reference, and drops conversational padding such as "can you tell me". It does not answer or summarize. Anything the model introduces that history does not support — a product name, a policy number, a date — is a new entity in your query, and it will match documents the user never meant (System Prompt Design covers the same discipline on the answer side).

The rewrite is one model call whose failure modes are all detectable before retrieval runs, which is what makes a fallback possible:

import json
from openai import OpenAI

client = OpenAI()

REWRITE = """Rewrite the user's latest message as a standalone search query.
Resolve pronouns and omitted nouns from the conversation. Add no facts.
Return JSON: {"query": str, "facets": [str, ...]} with 1-3 facets."""

def rewrite(history: list[dict], latest: str) -> dict:
    turns = "\n".join(f"{m['role']}: {m['content']}" for m in history)
    try:
        reply = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "system", "content": REWRITE},
                      {"role": "user", "content": f"{turns}\nlive: {latest}"}],
            response_format={"type": "json_object"},
            temperature=0,
        ).choices[0].message.content
        plan = json.loads(reply)
        if not plan.get("query", "").strip():
            raise ValueError("empty query")        # parses cleanly, still unusable
    except Exception:                              # timeout, 429, non-JSON, empty query
        return {"query": latest, "facets": [], "rewritten": False}
    return {"query": plan["query"],
            "facets": plan.get("facets", [])[:3],
            "rewritten": True}

Input:

history = [
    {"role": "user", "content": "Compare Bronze and Restore Plus for physical therapy."},
    {"role": "assistant", "content": "Bronze caps PT at 12 visits a year; Restore Plus has no cap but needs a referral."},
]
rewrite(history, "and what about the second one?")

Output:

{"query": "Restore Plus physical therapy coverage: visit cap and referral rules",
 "facets": ["Restore Plus physical therapy visits", "Restore Plus referral requirement"],
 "rewritten": true}

The pronoun resolved to Restore Plus because history named it; the facets split the two subjects the user cares about. The except branch is the whole reliability story: a timeout, a 429, malformed JSON, or an empty query all return the raw turn with rewritten: false, so search proceeds and the flag records how often the enhancement was skipped.

Multi-part questions need more than one retrieval: "what is the return window and who pays shipping?" names two subjects, and one query embedding averages them into a compromise that answers neither. Splitting into one facet per subject gives each its own search — but the result sets now come from separate queries, so their scores sit on different scales and cannot be compared directly. Merging them needs rank-based combination, which is exactly what Reciprocal Rank Fusion does.

Rewrites are chosen, not assumed. Score each configuration over the same frozen set of labeled queries before shipping one:

(40 labeled queries; illustrative.) The winner is the best row on your own set, and it stays the winner only until the corpus changes — re-run when documents are re-indexed (Evaluating Embedding Quality covers building the set).

When not to rewrite:

• Exact identifiers — E_CONN_TIMEOUT, order 88213, SKU NW-2210. Paraphrasing the one string that must match literally guarantees a miss.
• Operator syntax — level:ERROR svc:billing is already a query language; a model asked to "clean it up" will remove the operators.
• Hard latency budgets — the rewrite adds a full model round trip before retrieval begins, so a 300 ms end-to-end budget may not contain it.
• Saturated recall — if the raw query already lands the right passage in the top ten nearly always, the pass buys nothing and adds a failure surface.

Failure modes to log for:

• Intent drift — the rewrite answers a neighboring question; the eval set catches it, eyeballing does not.
• Hallucinated entities — a product name or policy number that appears nowhere in history.
• Uncapped facets — the model returns seven; the code slices to three and logs the truncation.
• Silent fallback — rewritten false on every request means the stage is dead and nobody noticed.