Capstone: Production RAG Assistant

A notebook RAG prototype answers questions from three PDFs. It dies the moment real users arrive: ingestion silently drops tables, retrieval returns overlapping chunks, answers cite nothing, and nobody can tell when quality degrades. The capstone exists to close those gaps — not to add features, but to make each stage observable, replaceable, and safe.

The system has two paths. An offline path runs on a schedule: parse documents, chunk them, embed the chunks, and upsert them into a vector store with metadata. An online path serves each user request: retrieve candidate passages, rerank them, assemble a token-budgeted context, and generate an answer that cites its sources. A third component — the eval harness — scores the whole system against a golden question set so regressions surface in CI, not in production.

┌──────────────────────────────────────────────────────────────┐
│                  CAPSTONE RAG ASSISTANT                      │
├──────────────────────────────────────────────────────────────┤
│  OFFLINE (scheduled)          ONLINE (per request)           │
│                                                              │
│  1. INGESTION                 4. RETRIEVAL                    │
│     parse PDF / HTML             embed query → top-k         │
│     chunk + overlap              hybrid: vector + BM25       │
│     embed → upsert                          │                │
│     metadata: source,                       ▼                │
│     page, section             5. RERANK (cross-encoder)      │
│         │                        top-k → top-5               │
│         ▼                                   │                │
│  2. VECTOR STORE                          ▼                │
│     chunks + metadata         6. CONTEXT ASSEMBLY           │
│                                dedupe, budget, order         │
│  3. EVAL HARNESS                          │                │
│     golden Q&A set             7. GROUNDED GENERATION        │
│     recall@k, faithfulness       answer + [citations]        │
│                                            │                │
│                                            ▼                │
│                              8. GUARDRAILS + REVIEW          │
│                                 PII redaction, injection     │
│                                 filter, human review queue   │
└──────────────────────────────────────────────────────────────┘

Design rules that keep this maintainable: each stage is a replaceable module, so you can swap the embedding model without touching generation; ingestion is idempotent, so re-runs update changed documents instead of duplicating them; every stage emits metrics, so a quality regression is attributable to a stage, not a mystery. For the stage-by-stage map, see RAG Pipeline Overview; for ingestion depth, see Building the Ingestion Pipeline.

Naive text extraction loses structure — headings, tables, and section boundaries vanish — and re-ingesting the same corpus creates duplicate chunks that double-count every answer. The ingestion pipeline fixes both: parse for structure, chunk with overlap, and derive stable IDs so re-runs upsert instead of duplicate.

import hashlib

def chunk_text(text: str, chunk_size: int = 512, overlap: int = 64) -> list[dict]:
    # Split text into overlapping chunks with stable IDs for idempotent upserts.
    chunks = []
    step = chunk_size - overlap
    for start in range(0, len(text), step):
        body = text[start : start + chunk_size].strip()
        if not body:
            continue
        chunk_id = hashlib.sha1(f"{start}:{body[:80]}".encode()).hexdigest()[:16]
        chunks.append({"id": chunk_id, "text": body, "start": start})
    return chunks

Two lines carry the design. step = chunk_size - overlap keeps 64 tokens of context across a boundary, so a sentence split between chunks survives in at least one of them. The ID is derived from position and content, not from a counter — re-running ingestion on an unchanged document produces the same IDs, so the vector store upserts rather than appends.

Embedding and upserting attach the metadata that citations depend on:

def upsert_chunks(chunks: list[dict], source: str, embed_fn, vector_store) -> None:
    for c in chunks:
        vector_store.upsert(
            id=c["id"],
            vector=embed_fn(c["text"]),
            metadata={"source": source, "start": c["start"]},
        )

Input: a 40-page PDF and 12 web pages. Output: 312 chunks, each carrying its source URL, page number, and section title. When a page changes, its content hash changes, only that page is re-embedded, and the rest of the index is untouched. For chunking strategy tradeoffs, see Fixed-Size Chunking and Semantic and Structural Chunking.

Top-k vector search alone returns overlapping chunks that waste the context budget, and when retrieval is thin the model quietly answers from memory — confidently, and without citations. The online path fixes this with a two-stage retrieve-then-rerank design, a hard token budget, and citation IDs the model must use.

def answer_question(question: str, embed_fn, vector_store, reranker, llm) -> dict:
    query_vec = embed_fn(question)
    candidates = vector_store.search(query_vec, k=20)                 # stage 1: recall
    top = reranker.rerank(question, [c.text for c in candidates], top_n=5)  # stage 2: precision

    context_parts, used, budget = [], [], 3000
    for c in top:
        if budget - c.tokens < 0:
            break                    # hard cap: lowest-ranked passages are evicted first
        context_parts.append(f"[{c.id}] {c.text}")
        used.append(c.id)
        budget -= c.tokens

    prompt = (
        "Answer using only the passages below. Cite each fact as [id]. "
        "If the passages do not say, reply 'Not covered'.\n\n"
        + "\n\n".join(context_parts)
        + f"\n\nQuestion: {question}"
    )
    return {"answer": llm.complete(prompt), "citations": used}

Three lines do the real work. k=20 then top_n=5 exists because cross-encoders are far more accurate than embeddings but too slow to score a full corpus — the cheap first stage narrows the field, the expensive second stage orders it. The budget loop evicts lowest-ranked passages first, so the most relevant signal survives when the context window fills. And [c.id] markers let post-processing verify that every citation in the answer exists in the retrieved set — a cheap check that catches fabricated sources. For the reranker itself, see Reranking with Cross-Encoders; for context ordering, see Assembling Context for the Model.

A confident wrong answer is worse than a refusal, and you cannot improve what you cannot measure. The eval harness scores every change against a frozen golden set before it reaches production, and guardrails wrap the generation stage so a hostile input or a PII leak is caught at the boundary.

def run_eval(questions: list[dict], answer_fn, judge) -> dict:
    results = []
    for q in questions:
        out = answer_fn(q["question"])
        results.append({
            "id": q["id"],
            "recall": q["relevant_chunk_id"] in out["citations"],
            "faithfulness": judge.score(q["question"], out["answer"], q["gold_answer"]),
        })
    return {
        "recall@k": sum(r["recall"] for r in results) / len(results),
        "faithfulness": sum(r["faithfulness"] for r in results) / len(results),
    }

Input: 200 golden questions, each with a relevant chunk ID and a gold answer. Output: recall@k 0.87, faithfulness 0.91. The first number says the right passage is retrieved 87% of the time; the second says the answer stays faithful to it 91% of the time. When those numbers move, the stage that moved is obvious. For harness design, see Offline Evaluation Harness; for retrieval scoring, see Evaluating Embedding Quality.

Guardrails sit at both edges of the request. On input, a PII redaction pass strips emails and phone numbers before the prompt is built, and an injection filter screens for instruction-like patterns. On output, flagged conversations — injection attempts, low faithfulness, user reports — land in a human review queue rather than disappearing into logs. See Input and Output Guardrails and Human Review Workflows. With evals, dashboards, and guardrails wired in, the capstone is a system you can operate, not a demo you can only hope works.