ROC Curve and Threshold Selection

You have a trained classifier and one confusion matrix — but that matrix is an artifact of a single threshold, usually 0.5 by habit. The ROC curve is what you get when you stop being superstitious: sweep the threshold across every score the model produced and plot two rates at each cutoff. Work it on ten labeled examples, five positive, five negative, with scores:

positives: 0.95  0.85  0.70  0.45  0.20
negatives: 0.80  0.60  0.35  0.25  0.10

Two rates drive everything. TPR (true positive rate, sensitivity, recall) = TP / (TP + FN): what fraction of the five positives you caught. FPR = FP / (FP + TN): what fraction of the five negatives you wrongly flagged. Recall from Confusion Matrix, Precision, and Recall that TPR is just recall; FPR is precision's neglected cousin, and the ROC plot pairs them precisely because one normalizes by actual positives and the other by actual negatives.

Reading the table teaches the mechanism. At threshold 0.90 the model flags only its most confident positive: TPR 0.2, FPR 0 — the point (0, 0.2) on the plot. Drop to 0.70 and you catch three of five positives while the one high-scoring negative (0.80) is the only casualty: (0.2, 0.6). At the bottom, threshold 0.10 flags everything, giving TPR 1 and FPR 1 — the unavoidable corner (1, 1). Every threshold is one point; connecting them in order of decreasing threshold is the ROC curve. A perfect classifier visits (0, 1) — all positives ranked above all negatives — hugging the top-left corner. A random scorer traces the diagonal y = x, because catching half the positives requires flagging half the negatives.

Two structural facts round out the construction. The curve is a step function: it only moves when the threshold crosses an observed score, so a model with 10,000 training rows produces a finely resolved curve while this toy example gives eight coarse steps. And the endpoints are fixed regardless of model quality: every ROC curve starts at (0, 0) and ends at (1, 1), so models are compared by how far they bow toward (0, 1) — which is exactly what the area under the curve quantifies.

In scikit-learn the sweep is one call:

from sklearn.metrics import roc_curve, roc_curve_display

fpr, tpr, thresholds = roc_curve(y_test, model.predict_proba(X_test)[:, 1])
# thresholds[0] is np.inf by convention so the curve starts at (0, 0)
roc_curve_display.from_estimator(model, X_test, y_test)  # plot in one line
# pick a point: idx = (tpr >= 0.90).argmax(); operating_threshold = thresholds[idx]

Notice the input: the ROC machinery consumes scores or probabilities, never hard labels. Averaging accuracy over a few thresholds would still hide the trade-off; the full curve exposes every operating point the trained model can reach — without ever retraining it.

The area under the ROC curve has a probabilistic meaning that makes it click: AUC is the probability that a randomly chosen positive example receives a higher score than a randomly chosen negative one. It is a rank statistic, not a calibration statistic, and you can verify it by brute force on the ten-example set from the last chapter. Five positives against five negatives gives 5 × 5 = 25 pairs to compare. Score 0.95 outranks all five negatives: 5 pairs. Score 0.85 also outranks all five (the best negative is 0.80): another 5. Score 0.70 beats 0.60, 0.35, 0.25, 0.10: 4. Score 0.45 beats three: 3. Score 0.20 beats only 0.10: 1. Concordant pairs total 5 + 5 + 4 + 3 + 1 = 18, so AUC = 18/25 = 0.72, which is exactly what roc_auc_score(y_test, scores) returns for this data.

The scale reads directly: AUC = 1.0 means every positive outranks every negative; AUC = 0.5 means the ranking is a coin flip — a random-scoring model scores 0.5 in expectation, which is why 0.5 is the useless-model baseline and why an AUC barely above 0.5 deserves a shrug, not a launch. AUC = 0.3 would mean the model ranks systematically backwards — often a swapped label encoding during evaluation, a bug worth checking before despairing.

Because AUC depends only on ranking, any strictly monotone transform of the scores leaves it unchanged. Squaring every probability, taking its logit, or rescaling to dollars moves every ROC point not at all; it only relabels the axis of thresholds. That invariance is the source of both AUC's power and its blind spot: AUC cannot tell you whether 0.9 means 90% or 40% chance — it does not measure calibration — and it averages over operating points you may never use (a system running at FPR 0.4 is rarely deployable, so the left tail of the curve often matters more than the whole-area summary).

That last point is sharpest under class imbalance, which is where ROC quietly flatters you. Take 10,000 transactions with 500 fraudulent (5%). A model operating at TPR 0.8 and FPR 0.04 looks excellent on the ROC plot — point (0.04, 0.8), far above the diagonal — and its ROC AUC might be 0.95. Now count people: TPR 0.8 catches 400 frauds, but FPR 0.04 means 4% of 9,500 clean transactions = 380 false alarms. Precision = 400 / 780 ≈ 0.51: half the review queue is noise, and if analysts clear 200 alerts a day, most of their day is wasted. The ROC did not lie — FPR is genuinely small — but its denominator is the 9,500 negatives, so a tiny FPR is still a large absolute crowd. The precision-recall curve, whose axes are both positive-class quantities, makes that pain visible: it starts at precision = 0.05 (the positive rate — the baseline to beat) and stays high only while precision holds as recall grows. Report ROC AUC for balanced problems and overall ranking quality; report PR AUC (average precision) whenever positives are rare and precision at the operating point is what the business actually pays for.

from sklearn.metrics import roc_auc_score, average_precision_score

roc_auc_score(y_test, scores)          # 0.95 — looks great at 5% positives
average_precision_score(y_test, scores)  # 0.40 — sobering; random baseline is 0.05
# always compare AP against the positive rate, not against 0

The curve and its area describe the model. The threshold is a business decision layered on top — and it can be optimized from costs, which is the mechanism of the next chapter.

Your model is trained and frozen, yet the product owner wants twice the fraud recall by Friday. Retraining is off the table — and it is unnecessary, because the threshold is a free parameter. The question is how to set it from costs instead of vibes, and the clean derivation is worth rehearsing.

Assume the model outputs calibrated probabilities — p is the true chance the row is positive, an assumption logistic regression's cross-entropy training makes reasonable. Two actions, two costs. Predict positive and the truth is negative: you pay the false-positive cost C_FP. Predict negative and the truth is positive: you pay the false-negative cost C_FN. The expected cost of flagging is C_FP × (1 − p); the expected cost of clearing is C_FN × p. Flag whenever the first is smaller:

C_FP (1 − p) < C_FN p   ⟺   p > C_FP / (C_FP + C_FN)

The optimal threshold is t = C_FP / (C_FP + C_FN).* Sanity-check the extremes: equal costs give t* = 0.5, the habit you were taught. If a missed fraud costs 19 times a false alarm (C_FN = 19, C_FP = 1), t* = 1/20 = 0.05 — flag anything above a 5% chance. If false alarms are expensive (a wrongful account freeze, C_FP = 19; C_FN = 1), t* = 0.95. The threshold is where the model's probability meets the cost ratio, and saying that sentence in an interview separates you from someone who tuned until the demo looked good.

import numpy as np
from sklearn.metrics import confusion_matrix

C_FN, C_FP = 19.0, 1.0
t_star = C_FP / (C_FP + C_FN)                    # 0.05
pred = (proba >= t_star).astype(int)
tn, fp, fn, tp = confusion_matrix(y_val, pred).ravel()
# or sweep and score each candidate directly:
for t in np.quantile(proba, np.linspace(0.01, 0.5, 50)):
    cost = C_FP * (proba[y_val == 0] >= t).sum() + C_FN * (proba[y_val == 1] < t).sum()

Three caveats keep this honest. Calibration first: the formula consumes probabilities, so if the scores are uncalibrated (frequently after class weighting or resampling — the subject of Class Imbalance Remedies), calibrate on a validation split (Platt or isotonic) before trusting t*, or tune the threshold empirically on the validation sweep shown above. Youden's J — maximize TPR − FPR — is the other famous rule; know its assumption, that it implicitly weights TPR and FPR equally and ignores prevalence, which is wrong exactly when the cost story is asymmetric. Validation discipline: pick t on validation data and report it once on the untouched test set; choosing the threshold on the test set leaks labels into the operating point and inflates every number you quote.

One more production concern worth raising unprompted: a threshold is a stored number tied to a score distribution, and both drift. If the model is retrained or the input distribution shifts, scores migrate and yesterday's t* silently changes the operating point — teams that retrain models but freeze the cutoff inherit a moving confusion matrix. Recompute the threshold whenever the model or the calibration changes, monitor the flagged-rate (predicted-positive share) as a cheap drift alarm, and prefer a small number of reviewed operating points over a continuous dial so every deployment state is auditable.

Also know what the threshold cannot fix. Moving it slides you along the model's fixed curve — trading precision for recall, never improving both at once. If no point on the curve satisfies the business (recall 0.9 at precision 0.6 when the curve tops out at 0.35), the model itself is the problem: more features, different training data, class reweighting. The curve tells you which of the two conversations you need to have, and that is the real value of plotting every cutoff instead of one.