ML System Design Interviews

The prompt appears on the whiteboard: design a system that recommends videos to 200M users. Forty-five minutes, no code. Most candidates open with a model — "I'd use a two-tower neural net with in-batch negatives" — and the interviewer's face closes. The model was never the question. An ML design interview tests whether you can move from a fuzzy business ask to a data source, a label, a training loop, a serving path, and a way to know it still works.

Run the answer in fixed segments, and spend the first segment on questions, not architecture.

A workable 45-minute budget: clarify the problem 4 min, labels and data 8, features 7, model 6, training loop 5, serving 8, evaluation and monitoring 5, risks and interviewer questions 2. If you reach "model choice" before minute 15, you are probably building on unexamined labels.

Concretely, the opening sounds like this: "The score drives which tile a user sees next, so ranking quality is the decision. Labels come from watch time and completes, delayed by up to a day — that delay shapes my retraining cadence. Before I pick a model: do we have logged impressions for every candidate item, or only for shown ones?" That last question alone separates candidates, because it decides whether your training data reflects user preference or the previous ranker's choices.

Two habits earn credit. First, name the prediction problem, the label, and the decision it drives before you name a model — everything downstream is a consequence of those three. Second, quantify one thing early ("200M DAU, say 3B watch events a day") so the interviewer knows you will do the math rather than hand-wave it. The estimation itself comes next; the framework's job is to make its inputs explicit and to leave room at the end for the two answers interviewers probe hardest: monitoring, and what happens when the model's own output poisons its future training data.

End the segment by restating the plan out loud — "I'll estimate storage and QPS, sketch training versus serving, then cover feedback loops and monitoring" — so the interviewer can steer instead of interrupt. A stated agenda turns the session into a conversation you are running.

The agenda is also your recovery plan. When the interviewer cuts across at minute 12 with "what about latency?", you answer from the serving column and return to your place in the plan, instead of losing the thread and restarting the whole design — candidates who never wrote the structure down derail every time the conversation branches, which in a good interview it always does. If the clock runs short, cut model depth and never monitoring: an answer that ends with "here is what I would alarm on, and here is the rollback" still demonstrates lifecycle thinking, while one that dies inside the architecture section reads as unfinished no matter how elegant the two-tower diagram was. The rubric most interviewers carry privately is weighted toward the boring parts — labels, cadence, failure modes — precisely because candidates rehearse the exciting parts.

"Roughly how much storage and how many servers?" is where candidates stall. The skill is not arithmetic precision; it is turning a few stated volumes into the quantities that constrain architecture — and saying the scaling law out loud as you go.

Work the standard sequence on these assumptions: 200M daily active users, 3B watch events per day, 100 float features per user profile, 90 days of training history, p99 serving budget of 50 ms.

Request rate. 3B / 86,400 ≈ 35,000 events/s average; assume 3× peak ≈ 100,000 rps. Recommendation retrieval typically runs offline or async per session, but the ranking call in the request path inherits that peak.

Feature storage. 500M profiles × 100 features × 4 bytes = 200 GB raw. A key-value store adds roughly 2× for indexes, replication, and overhead, so plan ~400 GB — shardable across a handful of instances, with attention to hot keys from viral content rather than to raw capacity.

Training data. 90 days × 3B = 27B rows. At ~400 bytes per row (100 features plus label and keys) that is ~11 TB raw; Parquet with columnar compression lands near 2 TB. The standard move is sampling: keep every positive interaction (~1% of rows) plus a 5% negative sample → roughly 1.5B rows, ~600 GB, which trains comfortably and preserves the signal.

Training compute. 1.5B rows through boosted trees takes hours on 16 cores; a deep ranker streaming 5M rows/s across 8 GPUs finishes an epoch in ~5 minutes, so a nightly full retrain with hourly incremental refresh fits any sane window.

Then the serving math, which most affects the design:

Throughput per replica. Batch-1 inference at 12 ms gives ~83 rps per replica — useless at 100k rps. Batching to 32 requests at 30 ms gives 32 / 0.03 ≈ 1,067 rps, a 12× throughput gain paid for in tail latency. Replicas needed at 70% target utilization: 100,000 / (1,067 × 0.7) ≈ 135 replicas. That fleet size is exactly where Serving Trade-offs: Latency and Cost earns its keep: 135 GPUs versus 135 CPUs, a bigger batch, or a distilled model are different bills for the same traffic, and the latency budget decides which of them is even legal.

A compact script keeps the arithmetic auditable instead of improvisational:

events_per_day, peak_factor = 3_000_000_000, 3
avg_rps = events_per_day / 86_400
peak_rps = avg_rps * peak_factor

profiles, feats, dtype = 500_000_000, 100, 4
raw_gb = profiles * feats * dtype / 1e9

batch, infer_s = 32, 0.03
per_replica = batch / infer_s
replicas = peak_rps / (per_replica * 0.7)
print(f"{peak_rps:,.0f} rps | {raw_gb:.0f} GB raw | {per_replica:,.0f} rps/replica | {replicas:.0f} replicas")

Two probes are worth inviting, because they separate arithmetic from understanding. First: "traffic doubles — which number changes?" Walk the chain again: peak QPS doubles, so replicas double; per-user feature storage does not move, because the profile count per user is unchanged; training data grows only with the number of days you retain. Knowing which quantities scale with which variable is the actual test. Second: "where would you cut first?" — point at the slack line and the batch size: doubling the batch recovers throughput at the price of tail latency, and if the budget can be renegotiated from 50 ms to 70 ms, replicas drop before features do. Candidates who can say which constraint they would spend first sound like people who have owned a bill.

State assumptions, do the multiplication, round hard, move on. The interviewer is checking that your server count scales with the variable that actually grows — QPS, users, or rows — not that you divided correctly three times.

The demo model scores 0.91 offline and 0.74 in production. Nothing about the architecture diagram explains that gap; the causes live in the seam between the training pipeline and the serving path. Every strong ML design answer names that seam explicitly, because it is where deployed systems quietly fail.

The non-point-in-time join deserves its own answer because interviewers probe it directly: if you build training rows by joining "current user state" to historical labels, every feature carries information that did not exist at prediction time, and the model learns shortcuts. The fix is mechanical — join each label to the state as of that moment:

# state must be sorted by event_time; labels are attached to the state that existed then
train_rows = pd.merge_asof(
    labels.sort_values("label_time"),
    state.sort_values("event_time"),
    left_on="label_time", right_on="event_time",
    direction="backward",
)

Then state the invariant in one sentence: the features you train on must be computable at serving time from data that existed at prediction time. A feature store with offline/online consistency is the standard answer to enforce it — the offline table trains the model, the online store serves the identical computation, and the join logic is versioned once.

Feedback loops are the second required close, and the usual example is the ranker itself. Training on logged impressions teaches the model the previous ranker's exposure pattern: high-ranked items collect clicks, low-ranked items never get labels, and the model becomes increasingly confident about an increasingly narrow catalog — position bias hardening into feedback. Mitigations to name: randomize a small exploration bucket, apply propensity weighting to correct logged policies, evaluate counterfactually before promoting, and ship in shadow mode before canary traffic. Where the loop touches sensitive outcomes (lending, moderation), say so — interviewers reward recognizing that feedback can amplify harm, not just error.

Be concrete about the rollout sequence, because it is a standard follow-up: ship the new ranker in shadow mode first — score live traffic, write predictions only to a log, compare against the incumbent on the same requests — then a 1% canary with automatic rollback on guardrail breach, then a 5 → 25 → 100% ramp while the online experiment runs. Each stage answers a different question: shadow tests skew and latency under real inputs, the canary tests blast radius, the ramp tests the metric.

Then close the whole answer with the monitoring loop, completing the lifecycle: input-distribution drift tests against the reference snapshot, prediction-distribution and volume alarms, calibration re-checked once delayed labels arrive, and latency and error budgets held as guardrails — the same discipline covered in Production Monitoring and Drift Detection, here argued as an integral part of the design rather than an afterthought. Monitoring the loop itself deserves three named signals: exposure share per item bucket (is the catalog narrowing?), label coverage for unshown candidates, and score entropy over time. A candidate who can list those three is answering the question nobody else reaches: how would you know your own model is making your data worse? When the interviewer then turns to your capacity estimates, you will be ready for the derivation-heavy follow-ups in ML Interview Question Patterns.