Production Monitoring and Drift Detection

The dashboard says AUC 0.87, unchanged for six weeks. Then quarter-end losses arrive and you discover the model has been wrong for a month. The dashboard watched the wrong thing: with labels arriving 30 days late, "live AUC" is computed on predictions made a month ago, while the inputs shifted last Tuesday. You need drift signals you can compute today — and you must know which drift each signal can and cannot see.

There are three distinct failures that people collapse into the word "drift":

KS and PSI detect distribution shift in X — they cannot see concept drift. If inputs are unchanged but the mapping to the outcome broke, every feature test passes while the model decays. Only fresh labels reveal it, which is why input monitoring and label-based monitoring are separate systems, not one dashboard.

The two-sample Kolmogorov-Smirnov test compares a numeric feature's reference and live samples by the largest vertical gap between their empirical CDFs:

D = max_x |F_ref(x) − F_live(x)|

and at α = 0.05 the critical value is approximately:

D_crit = 1.36 · sqrt((n + m) / (n · m))

Work it: with n = 2,000 reference and m = 2,000 live rows, D_crit = 1.36 · sqrt(4,000 / 4,000,000) = 1.36 · 0.0316 ≈ 0.043. Observed D = 0.071 exceeds it → reject equality → drift. Now shrink both samples to 500: D_crit = 1.36 · sqrt(1,000/250,000) ≈ 0.086, and the same D = 0.071 passes comfortably. Nothing about the data changed — only the sample size. Small monitoring windows hide drift, so minimum sample requirements belong in your alert thresholds.

from scipy.stats import ks_2samp

res = ks_2samp(reference["txn_amount"], live["txn_amount"])
# res.statistic = D, res.pvalue < 0.05 means the distributions differ
if res.statistic > 1.36 * ((len(reference) + len(live)) / (len(reference) * len(live))) ** 0.5:
    flag("txn_amount drifted")

For binned or categorical features, population stability index buckets live data into the reference's deciles and sums a weighted log-ratio:

PSI = Σ (A_i − E_i) · ln(A_i / E_i), where E_i is the reference fraction and A_i the live fraction per bin.

A worked example with reference deciles (E = 0.10 each) and live fractions [0.04, 0.06, 0.08, 0.10, 0.12, 0.14, 0.14, 0.12, 0.10, 0.10]:

PSI ≈ 0.114. The conventional reading: below 0.10 stable, 0.10-0.25 investigate, above 0.25 strong shift that warrants the retraining path. So this feature lands in "investigate" — meaning someone looks this week, not a page at 3 a.m. Two implementation cautions: PSI needs enough samples per bin (under ~1,000 rows, empty bins make ln blow up — add smoothing or don't compute it), and KS at α = 0.05 false-alarms on 5% of tests by chance, which becomes a real problem when you monitor 60 features. That threshold design is the next chapter's subject.

Monitoring that requires someone to open a dashboard is not monitoring — it is a report nobody requested. Alerting is the working part, and it is a design problem: every alert needs a signal, a window, a threshold, a severity, and the piece teams skip — the action it triggers. An alert without a runbook entry becomes noise inside two weeks, and noise trains the on-call to ignore everything, including the page that mattered.

A starting baseline for a synchronous fraud endpoint with a p99 SLO of 200 ms:

Note the severity split: latency and errors are page (users are hurting now); drift and freshness are ticket (degradation, nothing is broken this minute). Paging on drift trains people to ignore pages; ticketing on a latency breach trains people to ignore drift.

Two disciplines separate working alerts from pager spam:

1. Sample size and multiple comparisons. The KS math from the previous chapter shows the critical value depends on sample size — comparing 200 random rows hourly will fire on noise. Require a minimum n (1,000+ rows per comparison) or aggregate to daily windows for slow features. Then correct for multiplicity: testing 60 features at α = 0.05 expects ~3 false alarms per day by chance alone. Fixes: screen at α = 0.01, require persistence (PSI > 0.25 three days running), and apply a Holm-Bonferroni correction if you alert per-feature rather than on a single worst-of-set statistic.
2. Compare like with like. Traffic, transaction amounts, and feature distributions are seasonal — Sunday is not Tuesday, and month-end is not the 15th. A threshold of "yesterday's value ± 40%" pages every Monday morning. Compare against the same window last week, or model a seasonal expected value and alert on the residual. The prediction-distribution row in the table exists for the same reason: a mean score 3σ off last Monday is signal; 3σ off yesterday is the weekend.

Watch the serving layer even when the model is statistically fine. GPU utilization (an idle GPU bills identically to a busy one), queue depth (it rises minutes before p99 breaches — your earliest latency warning), cache hit rate, and feature-store miss rate all predict user-visible failure earlier than the user-visible metric does. Queue depth leading p99 is the detail interviewers remember, because it shows you monitor causes, not just symptoms.

Finish the wiring an alert needs: every row maps to a runbook section that says confirm (is this real? check for a deploy, an incident, a schema change), classify (upstream bug → fix upstream; genuine shift → train a challenger; broken labels → fix labels first), and record the outcome so thresholds get tuned from evidence instead of opinion. Without that loop, the table above is decoration.

Concept drift lives where labels live: falling accuracy, rising log-loss, or worsening calibration on feedback data. Two properties of real labels make this much harder than "join predictions to outcomes" — labels arrive late, and labels arrive selectively. Get either wrong and your quality metric lies to you in the confident direction.

Late labels. A default is known 90 days after the loan; a churn outcome shows up weeks after the behavior. Build a daily job that joins newly arrived outcomes to the predictions they belong to, keyed by (entity, model_version), and compute metrics over a rolling 14-day window of those predictions. The version key is the part teams omit: once v7 deploys, old labels must keep scoring v6's predictions so each version's true quality curve stays visible. Join on prediction time, not on today — otherwise every version inherits the others' results.

Selective labels. A collections agency gets default labels only for loans it approved; rejected applicants' outcomes are never observed. Evaluate naively on observed labels and the model looks excellent — you are measuring "how did we do on the cases we chose," and retraining on that measure reinforces the selection. Two standard corrections: reserve a small randomized holdout (1-2% of traffic) whose decisions ignore the model, giving an unbiased window into reality, or apply inverse-propensity weighting to scale each observed label by the inverse probability it got labeled. Interviewers who know production ask this; naming it separates dashboard readers from ML engineers.

Encode retraining triggers instead of debating them weekly:

retrain_triggers:
  data_drift:  { psi_max: 0.25, sustained_days: 3 }
  quality:     { auc_drop_pts: 2.0, window_days: 14 }
  freshness:   { feature_age_hours: 4, features: [txn_agg_90d] }
  calendar:    { every_days: 30 }        # periodic backstop, drift or not
any_trigger: open_challenger             # trains a candidate — never auto-promotes

The last line is the safety property: a trigger opens a challenger, it never overwrites production. The challenger trains, then runs the exact evaluation gate from CI/CD for ML Pipelines — effect size, paired-bootstrap confidence, slice guardrails, canary — before it can touch a single live request. Monitoring detects; the gate decides. Collapsing the two is how a data glitch becomes a self-inflicted outage.

Every quality alert needs a three-step runbook, because "accuracy dropped" has three different causes with opposite fixes:

1. Confirm it is real. Check for a recent deploy, an upstream schema change, an incident, or a broken label feed. Roughly half of these alerts resolve to a pipeline problem, and retraining on bad labels makes the model worse, not better.
2. Classify. Input pipeline bug → fix upstream, no retrain. Genuine distribution shift → train the challenger (the calendar/drift triggers above). Broken or delayed labels → fix the label path first; you cannot measure quality with a broken meter.
3. Record the outcome. Note in the incident whether the threshold was right (too sensitive? too slow?) and adjust the numbers from evidence. Thresholds that never get tuned stay wrong forever.

A randomized 1% control group — kept on the previous model, decisions unchanged — is the cheapest insurance in this chapter. When control and candidate both degrade, the world moved (data or labels). When only the candidate degrades, your model is the problem. Without the control you are guessing between those two, and guessing is not monitoring.

This is the last operational layer of the lifecycle: endpoints serving under an SLO, gated promotions, and monitoring that turns degradation into a challenger run on evidence. The capstone asks you to wire all three together into one pipeline you can demo and defend: End-to-End Capstone Project.