Evaluating Embedding Quality

Every retrieval metric is an average over labeled query-to-document pairs, which makes the labels the instrument. A broken instrument produces confident nonsense — and the most common breakage is measuring the team's own vocabulary instead of the customer's.

Twelve queries, every one written by people who built the index, every one returning the right document. The demo passed, the feature shipped, and the first week of production produced tickets titled 'search returns nothing for our own part numbers'.

Queries written by the team flatter retrieval because the team already knows the corpus vocabulary. Real queries arrive abbreviated, misspelled, one clause long, phrased in the customer's words instead of the documentation's. Closing that gap is a measurement problem, and measurement starts with labels:

labeled set = [ (query, {relevant doc ids}), ... ]     # a few hundred pairs
        ▲                        ▲
  support tickets, search    marked by a person, blind
  logs, failed searches      to what the index returned

A few hundred pairs is enough to expose large quality gaps. On twelve queries, models recalling 0.92 and 0.71 are statistically indistinguishable — the ordering is luck. Over three hundred pairs the confidence interval narrows to a couple of points and the ranking means something.

Sources of labels, in ascending order of trustworthiness:

The subtle trap is labels marked after seeing the rankings. A judge who has already read the retrieved list tends to call the top result relevant, and the evaluation then certifies whatever the current model already does. Mark blind, freeze the set before tuning, and treat every production miss as a new case rather than a one-off complaint. How the set gets built decides what the numbers can say; Building Eval Datasets covers the pipeline in depth.

Two numbers cover most retrieval questions: did the right document make the list, and how high did it sit.

def recall_at_k(ranked, relevant, k):
    return len(set(ranked[:k]) & set(relevant)) / len(relevant)

def reciprocal_rank(ranked, relevant):
    for i, doc in enumerate(ranked, start=1):   # rank is 1-based
        if doc in relevant:
            return 1 / i
    return 0.0                                  # never returned: full miss

def evaluate(run, labels, k=5):
    r1, rk, rrs = [], [], []
    for query, relevant in labels.items():
        ranked = run[query]
        r1.append(recall_at_k(ranked, relevant, 1))
        rk.append(recall_at_k(ranked, relevant, k))
        rrs.append(reciprocal_rank(ranked, relevant))
    n = len(labels)
    return {'recall@1': round(sum(r1) / n, 4),
            'recall@5': round(sum(rk) / n, 4),
            'mrr':      round(sum(rrs) / n, 4)}

labels = {
    'refund timing':     {'policy-refunds', 'faq-returns'},
    'reset my password': {'acct-recovery'},
    'enterprise sso':    {'pricing-enterprise'},
}
run = {
    'refund timing':     ['policy-refunds', 'faq-shipping', 'faq-returns'],
    'reset my password': ['faq-login', 'acct-recovery', 'policy-privacy'],
    'enterprise sso':    ['faq-login', 'policy-refunds', 'pricing-enterprise'],
}
print(evaluate(run, labels))

Output:

{'recall@1': 0.1667, 'recall@5': 1.0, 'mrr': 0.6111}

Four decisions in that code carry the interpretation:

• / len(relevant) gives partial credit on multi-label queries: finding one of two relevant documents is 0.5, not success. Averaging per query afterwards keeps a query with four labels from counting as four queries.
• enumerate(start=1) because rank is 1-based — reciprocal rank of a rank-0 position would be a division by zero, and off-by-one here silently doubles every score.
• return 0.0 on a total miss rather than skipping the query. A miss is a zero, not a missing data point; skipping it inflates the average by shrinking the denominator.
• Averaging per query, not per pair. recall@k answers "for what share of questions did the right chunk enter the list" — the unit of experience is the query, not the label.

The point of the metrics is to make decisions with them. An illustrative run on 320 labeled pairs from one product's documentation:

Read it as three separate decisions. The tuned model buys 13 points of recall@1 for double the price — worth it when retrieval feeds answers, probably not worth it for a deduplicator. Truncation gives back 7 points of the tuned model's gain while shrinking the index sixfold, which is the dimension trade from Embedding Models Explained made concrete. And recall@5 moving only 5 points across all three rows says the top of the list is where the models differ, not the candidate set.

k is its own decision. Recall@k rises with k and so does the prompt: every extra chunk is tokens on every request, so pick k where recall flattens and pay for nothing beyond it (Context Window Budgeting).

Offline numbers are a floor, not the finish line. Retrieval can improve while answers get worse — chunks that are relevant but too fragmentary to support a sentence will raise recall@k and lower answer quality. Close the loop by running the whole pipeline over the same queries and grading the answers, with Offline Eval Harnesses for the mechanics and LLM-as-a-Judge for grading free text.

Failure modes of the evaluation itself:

• Judges who saw the rankings — relevance marks converge on what the current model returns.
• Selecting and reporting on the same set — the winning model's score is optimistically biased; hold a set back you never tune against.
• Judging only the top 5 while quoting recall@10 — the labels do not cover the list you are measuring.
• A number without its k, set size, and model string — "recall 0.87" is uninterpretable without all three.
• No retained failure archive — every regression is rediscovered from scratch.