AI System Design Interviews

The candidate who starts drawing boxes immediately has already lost the interview. The interviewer is not evaluating the diagram — they are evaluating how you scope. A system designed without constraints is a system you cannot defend.

Five questions anchor the scoping round:

The last two are the ones candidates skip, and they are the ones that separate a product system from a demo. A 200ms latency budget and a 2s budget produce different architectures; a $10 monthly ceiling and a $10,000 ceiling produce different architectures. State your assumptions explicitly — "assuming 100 QPS peak and a 2s budget" — so the interviewer can correct you, and so every later decision traces back to a number. For the broader interview context, see What is System Design?.

What skipping scoping costs you: ten minutes of drawing an architecture the interviewer then tells you is wrong for their constraints. The scoping round is not a formality — it is the cheapest correction mechanism in the interview. Interviewers reward candidates who state assumptions and invite pushback, because that is exactly what a senior engineer does in a real design review.

Once requirements are set, walk the data from document to answer without gaps. The interviewer is listening for stages you skip — and every skipped stage is a place the system can fail silently.

documents → [parse & chunk] → [embed] → vector store
                                              │
user query → [retrieve top-k] → [rerank] → [assemble context] → [generate] → answer + citations
                                              │
                        [trace + metrics] ←──────┘

Ingestion runs offline on a schedule: parse, chunk, embed, upsert with metadata. Retrieval and generation run online per request: embed the query, retrieve candidates, rerank, assemble a token-budgeted context, generate with citations. Monitoring wraps both paths: every stage emits latency, cost, and quality metrics, so a regression is attributable to a stage. The habit that matters here is naming each stage out loud as you draw it — "retrieval returns 20 candidates, the reranker cuts to 5" — because specificity is what the interviewer scores. For the stage map, see RAG Pipeline Overview.

Storage deserves a decision, not a mention. The vector store is chosen by scale and operational context: pgvector when you already run Postgres and the dataset is small, a managed service when you want zero operations at large scale. Name the tradeoff out loud — "we chose pgvector and accepted operational responsibility because the alternative cost 4x at our projected size" — because an unexamined default is indistinguishable from no decision.

"How would you know it works?" is the most common probe in an AI system design interview, and the answer is expected early — not as an afterthought. Name the offline metrics you would track before launch and the online metrics you would watch after.

The pattern to communicate: offline metrics gate every release — a change that drops faithfulness does not ship — while online metrics tell you whether the offline metrics were measuring the right thing. A system with only offline metrics drifts from user needs; a system with only online metrics cannot tell a real regression from noise. See Offline Evaluation Harness, LLM-as-Judge Techniques, and Online Metrics for AI Features.

The CI gate is what makes the offline metrics matter. A golden set that lives on someone's laptop catches nothing; the same set wired into CI fails the build when a prompt or model change drops faithfulness below threshold. Say the mechanism explicitly — "every pull request runs the eval harness, and a drop of more than two points blocks the merge" — because the interviewer is checking whether you have shipped software, not just designed it.

The final section is where candidates separate. Name concrete failure modes with specific mitigations — generic "we would monitor it" answers score zero.

Notice what each mitigation has in common: it limits blast radius. Injection is contained by permissioned tools, not by hoping the filter catches everything; an outage is absorbed by a fallback chain, not by retrying the same provider forever; a blowout is stopped by a spend cap, not by noticing the invoice. For depth on any row, see Prompt Injection Attack Patterns, Model Fallback and Degradation, and Budgeting AI Spend.

One more habit separates strong answers: quantify the tradeoff before defending it. "A reranker costs 150ms and buys us five points of precision — worth it under a 2s budget, not under 500ms" shows you reason in constraints. The interviewer can then probe either side of the tradeoff, and you have shown you can defend both.