End-to-End Capstone Project

An interviewer asks you to walk through your capstone "end to end." If your answer is "I downloaded a dataset, trained XGBoost, and got 0.93 AUC," the interview is effectively over: you described a homework assignment, not a system. End to end means you touched every stage a model passes through between a raw source and a decision — ingestion, a frozen dataset, a training pipeline, an evaluation against a baseline, a packaged artifact, a serving endpoint, and a signal that tells you the thing still works six weeks later.

Here is a six-week schedule that fits alongside a job. One artifact per week, one binary done-criterion.

The ordering is the point: every milestone ends in an artifact another person can run. Each week is not a block of "keep experimenting." If week 3 closes and nobody can execute python -m src.train --config configs/v1.yaml and get a model file, the milestone failed regardless of what the loss curve shows.

Week 1 is where capstones die quietly. Pulling data means freezing it: write the raw files to storage, record row counts and a content hash, and never mutate them again. Everything downstream — experiments, dashboards, drift alarms — points at that hash. Skipping it is why you cannot later answer "which data produced this number?", and it destroys the ground truth you would need for Production Monitoring and Drift Detection after launch.

Week 2's baseline should be embarrassing on purpose: majority class, a hand-written heuristic rule, logistic regression on two features. The baseline converts every later number from a vanity figure into a comparison. Log the run — config, git SHA, data hash, metric — because an untracked run cannot be reconstructed in an interview three months later.

Weeks 3 and 4 hold the modeling, but the real deliverable of week 4 is the slice table: performance per segment, per device tier, per geography — not one headline. Choose the operating threshold in week 4 as well, because the endpoint you ship in week 5 must answer with something: labels or scores. A score with no threshold silently pushes the decision onto the caller, and nobody owns it.

Week 5 is where latency stops being theoretical. Wrap the model behind a small HTTP service, hit it with a script firing your expected QPS, and record p50 and p99. Those numbers feed directly into Serving Trade-offs: Latency and Cost: whether batching, quantization, or a bigger instance is even on the table.

Week 6 wires the alarms — prediction distribution, feature freshness, error rate — writes the model card from this topic's final chapter, and ends with the rerun test: a clean machine, one command, metrics landing inside your declared tolerance. That test is the difference between a project you demoed and a project you can defend.

Your demo lands with 0.94 AUC on a churn model whose positive class is 3%. An interviewer nods and asks the only question that matters: "What changed because of that number?" If you cannot answer, the interview exposes the most common capstone failure — metrics chosen after the model, chosen to describe it rather than to judge it.

Write down three kinds of metric before any modeling code exists, and make each one a decision rule.

Two details do most of the work. First, the split must be temporal. A random split on transaction data lets the model train on December and test on November, and any time-ordered phenomenon — seasonality, a pricing change, a user's own trajectory — leaks across the boundary. Hold out the last two weeks and refuse to look at them until week 4.

Second, the online metric needs an owner and a decision. "Retention" is not yet a metric until you name the experiment: randomize at the user level (session-level randomization contaminates the arms when one person returns twice), hold out a control group, and pre-commit to how long it runs. For scale, a two-proportion test detecting a 2% relative lift on a 30% base rate at 80% power needs roughly 90,000 users per arm (n ≈ 7.84 · 2p(1-p) / δ²) — that number tells you the experiment runs for two full weekly cycles, not two days.

Capacity turns the offline metric into a business one. Your campaign team can call 5,000 customers a day. Sort the scores, take the top 5,000, and report precision at 5,000 alongside recall: at precision 0.42 you catch 2,100 genuine churners, versus 150 for random selection at a 3% base rate — a 14× lift that an executive understands instantly, where "PR-AUC improved 0.06" does not.

Guardrails need numbers too, and they need them written before the run so nobody can renegotiate them afterward. A defensible set for this project: p99 latency ≤ 50 ms at expected QPS, no more than 5,000 outbound contacts per day, calibration error ≤ 0.03 on the operating slice, and daily prediction volume within ±10% of the trailing week. Every guardrail gets an alarm and an owner — a guardrail nobody is paged for is decoration. Note what is deliberately absent: "accuracy in production" cannot be a guardrail, because delayed churn labels make it unmeasurable for weeks; guardrails must be computable from serving-time signals alone, or they fail exactly when you need them.

Turn the whole set into pre-registered acceptance criteria in the README: ship if PR-AUC ≥ 0.55 against the 0.31 baseline, precision@5,000 ≥ 0.35, all guardrails green for seven days, and the online test shows non-inferior retention. Pre-registration is the honest alternative to moving goalposts: when the model misses, the criteria tell you which lever to pull instead of inviting you to select a friendlier metric after the fact.

Interviewers ask "how would you know your model worked?" more often as a metrics question than a modeling question. A strong answer walks the table out loud: this offline number is the proxy, that online number is the decision, these guardrails are what I refuse to break — and here is the baseline each one is measured against.

A reviewer clones your capstone, runs the training script, and gets 0.81 where your README claims 0.93. Nothing in the repository says which of them is wrong. That is the failure this chapter removes, with two documents: a model card for humans and an experiment record for machines.

The model card is a short structured document — a page, not a wiki. Interviewers ask "how do you document a model for handoff?" and the sections are the answer:

The experiment record is what makes those claims checkable. Log every run with the config, git SHA, and data hash attached:

import hashlib, subprocess, mlflow

data_sha = hashlib.sha256(open("data/train.parquet", "rb").read()).hexdigest()[:16]
mlflow.set_experiment("capstone-churn")
with mlflow.start_run():
    mlflow.set_tag("git_sha", subprocess.check_output(["git", "rev-parse", "HEAD"]).decode().strip())
    mlflow.log_param("data_sha256", data_sha)
    mlflow.log_param("seed", 42)
    mlflow.log_metric("pr_auc", 0.63)
    mlflow.log_metric("precision_at_5k", 0.42)
    mlflow.log_artifact("reports/slice_metrics.csv")

Then run the rerun contract once, yourself, before you publish: a clean environment, one command, and a result that lands inside the tolerance you print in the README — ideally a mean over three seeds rather than a single lucky number. Only after that do you connect the endpoint to the drift and latency alarms you designed in Production Monitoring and Drift Detection, so the card's "monitoring" row points at something real.

The run record itself is small — five fields, every time:

With those fields, handoff conversations become concrete: the receiving engineer diffs two runs field by field and sees whether a metric move came from data, code, or configuration — the same three-way localization you will be asked to perform when an interviewer says "production accuracy dropped 6 points; investigate." The monitoring row of the card should name signals rather than the word "monitoring": input-distribution distance against the snapshot hash, prediction-volume and score-distribution alarms, calibration re-checked as delayed labels arrive, and latency and error budgets held as guardrails, each with a threshold and a destination — dashboard, pager, or retrain trigger.

A headline metric without a data hash and a code SHA is a rumor. The card plus the run record turns your capstone into a claim a stranger can verify in an afternoon — which is exactly the impression the capstone is meant to leave.