Reranking with Cross-Encoders

The assistant answered "the plan covers physical therapy in general" when the customer asked whether Restore Plus covers physical therapy after knee surgery. The paragraph that answers it — Post-operative PT requires a referral and is uncapped — sat at rank 37 in the vector index. Retrieval asked for ten, so generation never saw it.

Nothing was broken in the usual sense. A bi-encoder embeds the query and each passage separately and compares the two vectors afterwards:

bi-encoder (stage 1)                cross-encoder (stage 2)

 q --> [encoder] --> v_q             (query, passage) --> [transformer] --> logit
 p --> [encoder] --> v_p
 score = cos(v_q, v_p)              every query token attends to every passage token

 embed once, search millions        one forward pass per pair

The two texts never meet in stage 1, so lexical proximity does all the ranking work it can: the plan overview mentions coverage, plan, and therapy three times each and wins, while the clause says post-operative where the query says after knee surgery. The passage was recalled — it is in the candidate list — but ordered below nine passages that talk around it.

Recall and precision are separate jobs, and they are handed to separate stages:

1. Stage 1 must get the target inside the candidate set. If the clause sits at rank 37 and the set holds 10, no amount of rescoring helps.
2. Stage 2 must put it at the top. A cross-encoder reads query and passage jointly, so "post-operative PT requires a referral" and "after knee surgery" can agree — relevance computed with both texts in view.

Reranking is stage 2. It reorders candidates stage 1 produced, and its quality ceiling is exactly the recall of the list it was given.

The loop is short: build pairs, predict one score per pair, sort descending, take the top of the list.

from sentence_transformers import CrossEncoder

ranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def rerank(query: str, passages: list[str], top: int = 5) -> list[str]:
    scores = ranker.predict([(query, p) for p in passages])   # one forward pass per pair
    order = sorted(range(len(passages)), key=lambda i: scores[i], reverse=True)
    return [passages[i] for i in order[:top]]

The model returns raw logits — unbounded numbers whose scale means nothing on its own. Put them on a comparable footing before anything downstream reads them:

import math

logits = [-2.41, 1.18, -0.05, -3.90, 0.64]     # candidates arrive in stage-1 order c0..c4

def softmax(xs):
    peak = max(xs)
    exps = [math.exp(x - peak) for x in xs]     # subtracting the peak keeps exp() from overflowing
    total = sum(exps)
    return [e / total for e in exps]

probs = softmax(logits)
for i in sorted(range(len(logits)), key=logits.__getitem__, reverse=True):
    print(i, logits[i], round(probs[i], 3))

Output:

1 1.18 0.524
4 0.64 0.305
2 -0.05 0.153
0 -2.41 0.014
3 -3.9 0.003

The stage-1 order is not preserved: c1 takes the top slot, and c4 — last in the first-stage list — moves to second. That reversal is the entire product of the second stage.

Two things follow from the output. peak = max(xs) is subtraction for numerical safety, not style: without it, math.exp(1.18 - (-3.90)) on a larger logit set overflows to inf and every normalized score becomes nan. And the normalized column is the only one a threshold may read — raw logit scale shifts with the model, the layer, and the query itself, so "keep passages scoring above 0.5" is meaningless until the numbers share a range. Even then, calibrate the cutoff against labeled pairs from your own corpus rather than trusting the default.

Reranking buys precision with time, and the trade is only worth stating with numbers:

The depth of the candidate set is a measurement, not a convention. The failure in the first chapter — target at rank 37 — means a top-10 reranker would have been paid for and useless: it can only reshuffle the ten it was handed. Run your own labeled queries, compute recall@k for the values of k you are considering, and pick the smallest k where recall clears roughly 0.95. Everything past that point is latency spent re-scoring passages the first stage was already getting right.

Cost grows linearly in candidates, and each unit is a forward pass over the concatenated pair — which is why stage 1 exists at all, and why the reranker is capped rather than merely limited.

Failure modes worth naming in a design review:

• Reranking too shallow — top-10 with recall@10 at 0.70; the fix cannot see the target.
• Thresholds on raw logits — scale drifts per model and per query; normalize, then calibrate on labeled pairs.
• Passage truncation — the cross-encoder's maximum length cuts the clause that mattered; check tokenized length before trusting a low score.
• Latency stacked, never summed — rewrite + retrieve + rerank each look fine alone; measure the total against the budget (Latency Profiling).
• Domain mismatch — a reranker trained on web relevance labels applies its own idea of "relevant" to legal or medical text; re-evaluate on your own pairs before blaming the retriever.