Communicating Results to Stakeholders

You have eight minutes with the director of operations and you open with AUC improved from 0.89 to 0.91. The room disengages, because ROC-AUC is a ranking property of a probability distribution - it has no units anyone budgets in, and a 0.02 delta does not say what changes on Monday. The translation chain has three links, and skipping any one loses the room.

Link one: pick the operating point. Not the default 0.5 threshold - the one implied by costs and policy. A review costs an analyst four minutes; a missed fraud case costs about $180 in expected loss, so the useful threshold is where those costs balance, and you should be able to say where it sits and why.

Link two: convert to counts at real volume. At 40,000 transactions per day and a 0.15% fraud rate, roughly 60 fraud events happen daily. Recall at your threshold becomes a count of caught fraud; precision becomes a count of real work in the queue.

Link three: convert counts into hours and dollars - the units the stakeholder owns:

The worked arithmetic is the headline: the old rules flag 205 alerts per day, the model at the chosen threshold flags 123 and catches 43 of the 60. At four minutes per review, 205 alerts is 13.7 analyst-hours a day and 123 is 8.2 - about 5.5 hours a day returned to the team, roughly $70k a year, while catching more fraud than the rules did. AUC belongs in the appendix of that deck, not the title.

from sklearn.metrics import confusion_matrix, precision_recall_curve

prec, rec, thresholds = precision_recall_curve(y_val, proba)
t = 0.42                                    # chosen by review cost vs miss cost
tn, fp, fn, tp = confusion_matrix(y_val, proba >= t).ravel()
alerts = (fp + tp) * daily_volume / len(y_val)
hours = alerts * minutes_per_review / 60

Two rules keep this honest. Translate at a fixed operating point, because that is where humans and money sit - a threshold-independent metric cannot tell anyone what Monday looks like. And carry one number with its comparison and its measurement story (held-out quarter, threshold policy, confidence interval, from Model Cards and Documentation); the deep detail stays in the card, the talk holds one number.

The same chain has a negative direction you should be ready for: if the review asks what the model costs when it is wrong, translate that too. A 35% precision means 80 false alarms a day, each consuming an analyst's attention and, worse, training the team to distrust the queue - alert fatigue is a business cost with a number attached. Quantifying the downside of the operating point shows you understand the system you are changing, not just the metric you improved. And when the translation depends on assumptions (volume, review time, cost per miss), state them on the slide: the arithmetic is only as trustworthy as its inputs, and a stakeholder who disputes the four-minute review time should be able to swap in their own number and recompute.

A delta without a baseline is a number without meaning: +2.1 points invites the question over what?, and if you cannot answer it in one sentence, the rest of the deck is decoration. Four baselines, ranked by how much they persuade:

The production baseline deserves the most weight and gets the least attention in practice, because it means running the current system on your evaluation window: the rules engine's confusion matrix, the human queue's precision, last quarter's model's score. Stakeholders are not deciding whether your model is good in the abstract; they are deciding whether to replace something that already works, and only a side-by-side answers that.

Two disciplines make the comparison trustworthy. Compare on identical folds or the identical time window - you saw in Cross-Validation Strategies that split noise swamps small deltas, so run baseline and candidate on the same folds and bootstrap the fold-by-fold differences (or apply McNemar's test to paired predictions). Report absolute values, not only deltas: 0.61 to 0.63 is +2 points and still worse than the rules engine at 0.71; the delta slides while the decision does not change.

from sklearn.dummy import DummyClassifier
from sklearn.metrics import roc_auc_score

dummy = DummyClassifier(strategy="prior").fit(X_tr, y_tr)
print("dummy", roc_auc_score(y_te, dummy.predict_proba(X_te)[:, 1]))   # about 0.50
print("model", roc_auc_score(y_te, model.predict_proba(X_te)[:, 1]))  # 0.91

The dummy run is not theater: a pipeline that leaks the target, a label inversion, or a feature that silently became all-zeros will still produce a confident-looking model that cannot beat the dummy, and catching that before the launch meeting is the cheapest QA you will ever run. In interviews, naming DummyClassifier as your first baseline signals that you have been burned at least once.

Finally, weigh the cost of the win in the same slide. A +0.4-point model that needs a three-week migration, adds 40 ms to p99 latency, and puts a new service on the on-call rotation may lose to a rules tweak - stakeholders are buying net value, not metrics. The slide that earns trust shows baseline, candidate, delta with an interval, cost to adopt, and your recommendation. When the honest recommendation is ship the baseline, the ML adds nothing yet, saying it buys you the credibility to be believed next quarter.

One sequencing rule makes the whole slide defensible: order your baselines by cheapness and run them in that order. The dummy baseline costs minutes; if your model cannot beat it, the features or labels are broken and no expensive comparison matters. The heuristic baseline follows, then the production comparison, then the ablation for attribution. Each stage gates the next, so a stakeholder who sees the ladder understands not only what you beat but what you checked first - and an interview panel hearing my model scores 0.91 will almost always ask against what?, which is this ladder said out loud.

91.4% accuracy invites a commitment the number cannot keep. Your validation set is a sample of the population, so the true value is a distribution, and the professional move is to put that distribution on the slide:

import numpy as np
from sklearn.metrics import accuracy_score

def bootstrap_ci(y, pred, metric=accuracy_score, n_boot=2000, seed=0):
    rng = np.random.default_rng(seed)
    idx = rng.integers(0, len(y), (n_boot, len(y)))
    stats = np.array([metric(y[i], pred[i]) for i in idx])
    return stats.mean(), np.percentile(stats, [2.5, 97.5])

mean, (lo, hi) = bootstrap_ci(y_val, y_pred)
# 0.914, [0.901, 0.926]

The phrasing matters as much as the arithmetic: 91.4%, and we are 95% confident the true value lies between 90.1 and 92.6. That sentence reads as rigorous rather than shaky, and it protects you when the live number drifts. The same discipline applies to percentages in general: quoting 5.5 hours a day without the window it was measured over (a slow Tuesday?) invites the exact challenge you are trying to preempt. For a two-model claim, bootstrap the difference instead of eyeballing two intervals: candidate 0.914 versus baseline 0.909, with a paired difference interval of [-0.001, +0.011], means no measurable win yet - and saying so now is cheaper than walking a promised 0.5 back later. When the interval is wide, the honest fix is more evaluation data, not a narrower sentence.

The second half of the skill is omission. What you leave out of a stakeholder update:

• The metric you tuned on, presented as held out. State plainly which split selected the model and which one reported it.
• Single-seed best runs. Report the mean across seeds or folds; a lucky seed is a lottery ticket, not a result.
• Point estimates on tiny slices. n=38 gets insufficient data, never 0.87 with implied authority.
• The 20-metric dashboard. Lead with one decision-relevant metric plus two guardrails; everything else is appendix material.
• Cross-benchmark SOTA comparisons computed on different splits - they do not transfer, and someone in the room will know.
• Promises narrower than your interval. Will improve by 2.3 points is not supportable; expect 1 to 3 points, confirmed by a shadow run is.

One boundary deserves saying out loud in every review: the interval describes held-out historical data, not the live experiment that will decide the rollout. Name the next measurement step - shadow mode on live traffic, then a canary at a small percentage - and state the result that would change your mind. The credible speaker is not the one with the biggest number; it is the one who can say what the number does not cover. When the ranges are too wide to justify the change, the honest next move is more data or a better split - and the search that turns a wider margin into a win starts with Grid and Random Search.

Uncertainty also shapes the structure of the update itself. Open with the decision you are asking for, not the history of the project; put the metric, its interval, and the baseline comparison on the first slide; leave derivations, fold-level tables, and the 20-metric dashboard in an appendix you can open on request. Questions will test three things - whether the number is real (interval, baseline), whether it will hold live (shadow plan, monitoring), and what happens if it fails (rollback trigger). If your deck answers those three in order, the technical depth you carry as backup stops being a wall of slides and starts being evidence.