RAG Pipeline Overview

An HR support bot answered "The relocation cap for Berlin is $3,000" in a confident, well-formed sentence. The handbook sitting in its index said €4,000 plus a 90-day residency requirement. Nobody had put the handbook in front of the model for that question, so the model produced a number from what it had seen during training — plausible, specific, and wrong. An employee filed a reimbursement against it.

Retrieval-augmented generation (RAG) exists to make that class of answer structurally impossible rather than unlikely. It imposes two obligations on the system:

1. Put the right text in front of the model for each question — retrieved from your documents, not from the model's memory.
2. Make the answer come from that text — every factual claim traceable to a passage you actually sent.

Those two obligations map onto a fixed sequence of stages. Each stage takes a defined input, produces a defined output, and fails in its own characteristic way:

 OFFLINE — runs on a schedule              ONLINE — runs once per question
 --------------------------------          --------------------------------
                        +--------+
 documents ----------->| parse  |--+
                        +--------+  |
                                    v
                              +---------+
                              | clean   |
                              +---------+
                                    |
                                    v
                              +---------+        +--------+
                              | chunk   |------->| embed  |----> vector index
                              +---------+        +--------+
                                                          ^
  "what is the Berlin cap?"                               |
        |                                                 |
        +----> embed query --> retrieve top-k ------------+
                                          |
                                          v
                                assemble numbered context
                                          |
                                          v
                              generate answer, each claim [n]

Read the diagram left to right: the offline half converts files into vectors and can take minutes or hours; the online half must answer in the time a user will wait. Nothing on the right half writes to the index, and nothing on the left half ever sees a user question.

The retrieval stage may itself be two searches merged — keyword and vector — which is where fusion methods such as reciprocal rank fusion come in. The rest of this topic treats retrieval as one stage producing a ranked list of passages.

When a bot gives a bad answer, one of the two obligations broke. Learning to name which one is the whole job of this phase.

The most common architectural mistake is treating RAG as one synchronous job. The two halves run on different clocks, tolerate different failures, and expose different metrics.

Two consequences fall out of the table.

Failures repair on the clock that owns them. A missing document is an ingestion defect: re-run the job, fix the parser, backfill. Reprompting the model will not index the file. Conversely, a correct passage retrieved and then ignored by the model is a generation defect: no amount of re-chunking helps.

Offline work can be retried; online work cannot. The ingestion job may batch, resume, and re-run overnight. The query path has one budget — the user's patience — so every stage there is sized for latency: small k, a single embed call, one generation.

This split is also why index freshness matters. A handbook revised yesterday is not visible until the offline job runs again; until then the online path serves the old chunks with perfect faithfulness to stale text. Freshness is an ingestion metric, and it is the one most often missing from dashboards. The mechanics of that offline job — parsing, deduplication, provenance — are the subject of the ingestion pipeline topic.

Interviewers ask a sharper version of this: if quality drops 20% after a model upgrade, how do you know which stage regressed? The answer is the metric column: pull recall@k and faithfulness separately, because they implicate different halves of the diagram.

A pipeline earns its architecture when one line of the diagram can change without touching the others. The cheapest way to get that is to give every stage a narrow signature and pass plain data between them.

from dataclasses import dataclass

@dataclass
class Passage:
    pid: str          # stable id from ingestion, survives every re-rank
    text: str
    score: float = 0.0

def chunk(doc: str) -> list[str]:
    return [p.strip() for p in doc.split("\n\n") if p.strip()]

def embed(texts: list[str]) -> list[list[float]]:
    # stand-in for a real embedding call: counts of three known words
    vocab = {"berlin": 0, "cap": 1, "relocation": 2}
    out = []
    for t in texts:
        vec = [0.0] * len(vocab)
        for w in t.lower().split():
            if w in vocab:
                vec[vocab[w]] += 1.0
        out.append(vec)
    return out

def retrieve(index, qvec, k=2) -> list[Passage]:
    def sim(a, b):
        dot = sum(x * y for x, y in zip(a, b))
        norm = (sum(x * x for x in a) ** .5) * (sum(y * y for y in b) ** .5)
        return dot / norm if norm else 0.0
    scored = [Passage(pid, text, sim(vec, qvec)) for pid, text, vec in index]
    return sorted(scored, key=lambda p: p.score, reverse=True)[:k]

DOCS = [
    "relocation policy: the relocation cap for Berlin is 4000 euro",
    "expense rules: travel is reimbursed at 90 percent for economy fares",
    "handbook: parental leave is 14 weeks in Germany",
]

index = [(f"doc{i}", d, embed([d])[0]) for i, d in enumerate(DOCS)]
hits = retrieve(index, embed(["relocation cap Berlin"])[0])
print([(p.pid, round(p.score, 2)) for p in hits])

Output:

[('doc0', 0.94), ('doc2', 0.0)]

Three lines carry the design:

• Passage.pid is minted during ingestion and never recomputed, so a citation like [doc0] stays valid across embedding upgrades and re-rankers.
• embed takes a list and returns a list — batching is part of the contract, not an optimization bolted on later.
• retrieve only reads score for ordering; the moment a reranker changes the order, nothing else in the pipeline notices.

To upgrade, you replace the function, not the system: point embed at a hosted model with the same list[str] -> list[list[float]] shape, re-run ingestion, and the query path is untouched. That is what the blueprint means by designing each stage as a replaceable module — the model can improve on its own cadence while serving code stays frozen.

Where the pipeline goes wrong once it is assembled — and how to tell retrieval problems from generation problems — is the subject of diagnosing RAG failure modes.