ML Project Lifecycle

A logistics team asks you to predict parcel delivery times with machine learning. The request arrives as one sentence, and the temptation is to open a notebook, load the tracking database, and start fitting gradient-boosted trees. Six weeks later the model predicts delivery windows with decent RMSE, and the product team explains that what they needed was a promise they could print on the checkout page, with a penalty when the promise is wrong. The modeling work was not wasted because the wrong question was answered first.

CRISP-DM, the cross-industry standard process for data mining, exists to prevent exactly that. It sequences a project into phases that must complete in order, each with a concrete artifact:

The first phase forces three written answers. What decision changes because of this model (print a delivery promise at checkout)? How is success measured offline (mean absolute error of predicted vs actual delivery hours) and online (checkout conversion, promise-accuracy rate)? What is the baseline to beat (the current rule: always promise 3 days, or the courier's own estimate)? A model that beats a trivial rule is a result; a model that beats nothing is a demo.

Data understanding comes before any modeling because it usually kills or reshapes the project. Audit coverage (do rows exist for every warehouse and weekday?), label availability (do you have actual delivery times for enough historical parcels?), and candidate leakage columns (delivery_actual_duration obviously exists in the tracking database and must never become a feature). Thirty minutes of profiling here prevents the six-week wrong-answer scenario.

Phases 3 and 5 matter more than phase 4 for most teams. Feature preparation is where reproducibility is won: the same pipeline code that built training features must build serving features, or training-serving skew silently rots the model in production. And evaluation compares against the agreed business metric, not whatever loss the optimizer minimized: an MAE of 4 hours might pass if it beats the courier baseline by 30%, or fail outright if the business needed 2 hours to keep customers from contacting support. The next chapter takes apart the agreement that makes that comparison possible in the first place.

The single highest-leverage document in an ML project is one page long: the metric agreement. It is written with stakeholders before training starts, and it converts vague ambition into a number that can end the project, ship it, or force another iteration. Teams that skip this page discover at review time that data science measured ROC-AUC while the business cared about recall at a 5% alert budget, and that the model passes one bar and fails the other.

A complete metric agreement has three layers. The primary metric is the offline quantity that gates model selection: for the delivery project, mean absolute error in delivery hours on the chronological test set, with a stated threshold (beat the courier baseline by at least 20%). The online metric is what production A/B testing measures, because offline numbers only earn the right to be tested: promise-accuracy rate (fraction of parcels inside the promised window) and checkout conversion. The guardrails catch collateral damage: support-ticket volume, average promised window width (a model that promises 10 days is accurate and useless at the same time), and p95 latency of the scoring endpoint. Each layer names its owner, because a metric nobody watches is decoration.

# Delivery promise model - metric agreement
Primary (offline gate): MAE_hours on test, must beat courier baseline by >= 20%
Online (4-week A/B): promise-accuracy >= 78% AND checkout conversion non-inferior
Guardrails: p95 scoring latency < 80 ms; promised-window width not > baseline + 5%
Baseline: static 3-day promise, courier estimate v2
Kill criterion: if offline gate fails after 3 iterations, fall back to courier estimate fusion

Writing the kill criterion deserves special attention. Projects die by attrition when nobody defines failure, so the team keeps tuning long after the evidence says the data cannot support the question. Naming the failure condition upfront (three iterations without clearing the offline gate, or a data audit that shows labels for under 40% of the target population) turns an emotional debate into a checklist decision. It also protects the team from the sunk-cost trap: a documented kill is a successful experiment with a clear answer, not a wasted quarter.

The agreement also fixes the evaluation protocol from the previous topic. The offline gate names the split (chronological, last 8 weeks as test), the metric implementation (sklearn mean_absolute_error on delivery hours), and the number of test-set consultations allowed (one). With those pinned, disagreements during review become about the evidence rather than about which number each side chose after seeing the results. Stakeholders sign a one-pager; engineers stop relitigating; the project has a definition of done that a new teammate can read on day one.

Interviewers hear this topic as a process question: how do you run an ML project end to end? A strong answer names the phases, then dwells on the two artifacts most candidates forget, the written metric agreement with a kill criterion, and the baseline you committed to beating. That framing also sets up the honest answer about effort distribution, which the next chapter covers.

The metric sits at 71% against a 80% gate for two weeks. The default reaction is to try another algorithm, but swapping models is the most expensive and least likely fix. When progress stalls, the productive move is to diagnose which phase's artifact is weak and iterate there. Each stall pattern maps to a phase.

The row worth memorizing is the third one from the bottom. If logistic regression, random forests, and a small neural network all land at the same score, the bottleneck is the feature representation: no optimizer will extract signal the features do not contain. Teams that recognize this stop benchmarking models and spend the week on feature crosses, better embeddings, or a relabeled sample, which routinely buys more points than any architecture swap. This is what iterating back beats tuning algorithms when metrics stall means in practice: move backward one phase, repair the artifact, and only then continue forward.

Checkpoints formalize the loop. Each phase ends with a review that has exactly three outcomes: go (artifact meets its bar, proceed), iterate (diagnosable gap, return to the named phase with a time box), or kill (the data cannot support the question; document and stop). Time boxes matter because iteration without a deadline is how projects drift. A checkpoint needs evidence, not vibes: the EDA summary, the leakage review, the cross-validation table with spreads, the metric agreement, the A/B readout.

The final checkpoint before deployment compares the chosen model against the agreed primary metric and the baseline, and produces the handoff artifacts: a model card describing intended use and known failure modes, the pinned training code and data snapshot, the serving contract (input schema, latency budget, fallback behavior), and the monitoring plan with drift thresholds and retraining triggers. A model without a named fallback (serve the courier estimate when the endpoint times out) is not ready to ship.

For interviews, tell this story as a sequence with numbers: framed the problem with stakeholders, wrote the metric agreement and baseline, audited data and found a leakage column, tuned with chronological cross-validation, missed the gate once, went back to features, cleared it, shipped behind an A/B with guardrails, monitored drift. It demonstrates the whole lifecycle in one narrative, and it is exactly how the process is meant to run. Next you assemble the tools this workflow runs on in Python ML Data Stack.