Regression vs Classification Targets

Two projects land on your desk. The first asks for next month's demand per warehouse, answered in units with any value from 0 to 5,000. The second asks whether a payment will be disputed, answered yes or no. The first is regression and the second is classification, and the entire distinction lives in the target variable y. Nothing about the input decides it: the same customer table can serve a regression task (predict spend) and a classification task (predict will-spend-over-$500).

Regression targets are continuous numbers where distances mean something. Predicting price at $42,000 instead of $41,500 is a small miss; predicting $20,000 is a disaster. Because the error magnitude is meaningful, regression trains with squared or absolute error and reports metrics like MAE, RMSE, and R-squared. Classification targets are discrete labels where the identity matters, not the number. Predicting class 3 instead of class 4 of animal species is not halfway right in any useful sense, so the model trains with cross-entropy and reports accuracy, F1, or ROC-AUC.

from sklearn.linear_model import LinearRegression, LogisticRegression

# regression: y is a number with meaningful magnitude
X = [[1200], [1500], [1800], [2100]]       # square footage
y_price = [300_000, 360_000, 425_000, 480_000]
LinearRegression().fit(X, y_price).predict([[1900]])   # ~ 441,000

# classification: y is a label
y_churn = [0, 1, 0, 1]                     # stayed, left
LogisticRegression().fit(X, y_churn).predict([[1900]])  # 0 or 1

The failure mode interviewers probe is treating class labels as a regression target. Suppose you model delivery priority as 0 = standard, 1 = express, 2 = overnight and fit a regressor. It now assumes the distance from standard to express equals the distance from express to overnight, which nobody guaranteed, and it can output 1.4, a class that does not exist. Rounding the prediction back to 0, 1, or 2 hides the assumption but does not remove it. If the levels are genuinely ordered with uneven gaps, that is an ordinal problem, which you will handle as classification with an order-aware metric instead.

Count targets sit awkwardly in between. Predicting the number of support calls next week yields integers, but unlike class labels the magnitude is meaningful: missing by 40 calls hurts more than missing by 2. Teams usually regress on counts with RMSE, or use a Poisson loss that respects the non-negative integer nature of the target. Similarly, a bounded score between 0 and 1 (probability of default, click-through rate) is regression with a bounded output; logistic models handle it natively, while plain regression can wander outside [0, 1] unless you clip or transform.

The practical discipline: write the target column on the whiteboard first, list five example values, and ask two questions. Are the values discrete labels or numbers with meaningful distances? Would averaging two training targets produce a valid target? If yes to averaging (average demand of 300 and 500 units is 400 real units), you are in regression; if averaging produces nonsense (the average of spam and not-spam is not a class), you are in classification. Next you will see what a classification model actually emits before any label is assigned in From Probabilities to Decisions.

A classifier does not truly output a class. It outputs a score, usually a probability, and something else turns that score into a decision. Confusing the two layers is why engineers who come from regression mis-tune classification systems: they treat the predicted label as the model's output when the probability is the output, and the threshold is a business dial you set after training.

For binary problems the model emits a single number p between 0 and 1 through the sigmoid function applied to a linear score, and training minimizes cross-entropy against the 0/1 labels. Cross-entropy is the natural pairing for probabilities: it penalizes confident wrong answers far more than hesitant ones, which squared error on 0/1 does not do well. For multi-class problems the softmax layer turns k raw scores into k probabilities that sum to 1, and the argmax picks the label.

import numpy as np
from sklearn.linear_model import LogisticRegression

model = LogisticRegression().fit(X_train, y_train)   # y_train in {0, 1}
p = model.predict_proba(X_test)[:, 1]                # column for class 1

# default decision: label 1 when p >= 0.5
# fraud cost asymmetry: flag anything with meaningful risk
flags = p >= 0.10
print((p >= 0.5).mean(), flags.mean())               # e.g. 0.06 vs 0.31

The 0.5 cutoff is a convention, not a law, and moving it is free inference-time tuning. If a false negative costs $500 and a false positive costs $5 in analyst time, the optimal threshold drops far below 0.5; you flag more cases and review the queue manually. Two things must move together when you shift the threshold: the metric you report (precision falls as you flag more, recall rises) and the operating point you show stakeholders on the ROC or precision-recall curve. Precision-recall curves are the honest view when positives are rare, because ROC-AUC looks flattering even at extreme class imbalance.

The classic interview question is why not train a linear regressor on the 0/1 labels and threshold its output at 0.5. Three reasons. The regressor's output is an unbounded score with no probability semantics, so threshold sweeps lose their calibration meaning. Squared error on binary labels is dominated by the confident examples and punishes a hesitant but correct prediction almost as harshly as a wrong one, flattening the decision boundary where it matters. And the fitted line only approximates the class probabilities where the data is dense, drifting badly on outliers.

Pick the threshold on the validation set, never on the test set. A quick method: sweep thresholds from 0.01 to 0.99, compute expected cost at each (false positives times their cost plus false negatives times theirs), and take the argmin. On a calibrated model this expected-cost view is honest; on a poorly calibrated one you first fix calibration (Platt scaling or isotonic regression fit on validation predictions) because a threshold of 0.1 only means a 10% event rate if the probabilities are trustworthy.

Regression and classification also differ in what error looks like after the decision layer. A regression miss of $800 on a house price is graded continuously, and you inspect the residual distribution. A classification miss is only wrong when the threshold crosses the label, so you inspect the confusion matrix: which class absorbed the errors, and whether the model is calibrated. Checking calibration (reliability plots, Brier score) matters whenever probabilities themselves feed a downstream system, such as expected-loss calculations in pricing or ranking by risk.

Binary yes-or-no problems are the teaching case, but production targets come in three harder shapes: one label among many, several labels at once, and ordered levels with uneven spacing. Each looks superficially like the others, and choosing the wrong formulation silently corrupts both training and evaluation.

Multi-class means exactly one correct label per example, drawn from k options: which department receives a ticket (billing, access, shipping, other), which language a document is written in. The standard output is a softmax over k classes, or k one-vs-rest binary classifiers stitched together. Evaluation is standard accuracy plus per-class precision and recall, because a dominant class can hide total failure on a rare one.

Multi-label means an example can carry several labels simultaneously. A movie is comedy and romance and musical; a clinical note mentions diabetes, hypertension, and no cancer; a pull request touches auth, payments, and docs. Softmax is wrong here because it forces the k probabilities to compete for a total of 1. You need one sigmoid per label, k independent binary decisions, and metrics that count partial credit: per-label F1, macro-averaged F1, and Hamming loss (the fraction of labels predicted incorrectly). Exact-match accuracy, demanding the entire label set be perfect, is brutally strict and usually near zero on real data.

from sklearn.multiclass import OneVsRestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score

# y is a binary indicator matrix: each row may have several 1s
clf = OneVsRestClassifier(LogisticRegression()).fit(X_train, y_train_multilabel)
pred = clf.predict(X_test)                 # independent 0/1 per label
print(f1_score(y_test_multilabel, pred, average='macro'))
print((pred != y_test_multilabel).mean())  # Hamming loss

Ordinal targets are ordered levels where the ordering is real but the gaps are not uniform: star ratings 1 to 5, disease severity mild/moderate/severe, ticket priority low/medium/high/critical. Treating them as plain multi-class discards the order (predicting 5 instead of 1 looks as wrong as predicting 3), while treating them as regression invents equal spacing and emits values that are not classes. The clean approaches are cumulative (threshold) models that predict the probability of being above each level, or plain classification reported with an order-aware metric such as quadratic weighted kappa.

The last mixed case is count or bounded-score targets, which you met in the first chapter: counts keep meaningful magnitude, so regress on them with a Poisson or RMSE loss; bounded probabilities and rates get logistic-style outputs or clipping. One trap to name explicitly: multi-label exact-match accuracy demands the entire predicted set equal the true set, so predicting three of four correct topics scores zero. Report it for completeness if the business cares about perfect sets, but let macro F1 carry the signal, since it averages per-label scores and refuses to let a frequent label mask failures on rare ones.

The through-line for interviews and for practice: name the exact semantics of one target value (is order real? do gaps mean anything? can several labels co-occur?), and the loss, output layer, and metric follow mechanically. With targets settled, the next topic guards the estimate you will quote for all of them: Train, Validation, and Test Splits.