Multi-class Strategies

A support desk sorts tickets into four queues: billing, bug report, account access, and feature request. You already know binary logistic regression from Logistic Regression Fundamentals, and when you hand four-string labels to scikit-learn's LogisticRegression it simply fits. No error, no code change. The question an interviewer will ask next is the one that separates users from people who understand the model: what did fit actually build?

With multi_class='ovr' (or an explicit OneVsRestClassifier), it built K independent binary classifiers. For four classes you get "billing versus everything else", "bug versus everything else", "account access versus the rest", and "feature request versus the rest". Each one is the sigmoid model you already know, trained on the original features with a 1-vs-rest relabeling. Prediction takes the raw score of each classifier and picks the largest:

from sklearn.multiclass import OneVsRestClassifier
from sklearn.linear_model import LogisticRegression

ovr = OneVsRestClassifier(LogisticRegression(max_iter=1000))
ovr.fit(X_train, y_train)          # y in {billing, bug, access, feature}
scores = ovr.decision_function(X_test)   # shape (n_samples, 4)
pred = scores.argmax(axis=1)             # highest score wins

Three properties of this construction matter, and each one bites in production.

Concretely, one row of scores might come back as billing −1.8, bug +3.1, access +0.4, request −0.9. Bug wins the argmax, and the raw margins tell you something a probability never could: this row is a decisive bug report (+3.1 versus +0.4 for the runner-up) rather than a coin flip between two queues. Routing logic downstream can use that margin to auto-close confident tickets and escalate ambiguous ones.

First, the classifiers never talk to each other. Raising the bug classifier's score does not lower the billing classifier's score, so a row can score positive for two classes at once — the argmax silently breaks the tie, and a row can score positive for zero classes and still get a label. That independence is a feature when labels are genuinely not mutually exclusive: multi-label problems ("this email is about billing AND account access") are exactly K sigmoid problems, and softmax — the alternative below — mathematically cannot represent them because it forces the classes to compete for one unit of probability.

Second, the per-class scores are not comparable probabilities. Each sigmoid estimates P(class k | features), which is really the probability of a one-vs-rest event. For an easy row, all four sigmoids can confidently say yes, so the four probabilities might sum to 2.3. scikit-learn's predict_proba divides each by the sum to fake a distribution — a convenience normalization, not a model output. Calibration drifts per class too: the rare class's classifier is trained on a heavily skewed binary problem and its probabilities are systematically off.

Third, imbalalance is multiplied, not avoided. The garden-tools class with 40 examples in 10,000 rows faces a 40-vs-9,960 binary fight in its own classifier while the dominant class faces a nearly balanced one. The default 0.5 threshold then classifies almost nothing as garden tools, and its recall collapses toward zero even while overall accuracy looks fine. You will meet the full diagnosis of that failure in Class Imbalance Remedies; the point here is that OvR creates K differently-skewed training problems from one dataset.

One nice structural fact worth saying out loud in an interview: because the sigmoid is strictly monotone, argmax over the probabilities equals argmax over the raw logits, so the decision rule never depends on calibration — only ranking within each classifier. But do not threshold each sigmoid at 0.5 independently; that is a different rule, and it produces both double-claims and no-claim rows.

The other standard strategy throws away the per-class binary framing entirely. Multinomial (softmax) logistic regression keeps one weight vector w_k per class, scores all K classes at once, and normalizes them into a single distribution:

z_k = w_k^T x + b_k                      # one logit per class
softmax(z)_k = e^(z_k) / sum_j e^(z_j)   # probabilities that sum to 1

Work a three-class example by hand. The logits come out as [2.0, 1.0, 0.1]. Exponentiate: e^2.0 = 7.389, e^1.0 = 2.718, e^0.1 = 1.105, summing to 11.212. Divide: probabilities [0.659, 0.242, 0.099] — they sum to exactly 1, and the predicted class is the argmax. Note two mathematical properties that show up in interviews. The softmax is invariant to adding a constant to every logit (the constant cancels in the ratio), so only K−1 degrees of freedom are really identifiable. And it is monotone in each logit, so the predicted class is again just argmax of the raw scores.

That invariance is also the numerical survival trick. Feed logits like [1000, 999] to np.exp and you get inf, then inf/inf = nan. Subtract the max logit first — [0, −1] — and the probabilities come out identical and finite. This is the log-sum-exp stabilization, and every real implementation does it:

import numpy as np

def softmax(z):
    z = z - z.max()          # shift by max; cancels in the ratio
    e = np.exp(z)
    return e / e.sum()

print(softmax(np.array([2.0, 1.0, 0.1])))   # [0.659 0.242 0.099]
print(softmax(np.array([1000.0, 999.0])))   # [0.731 0.269], no overflow

Training minimizes multiclass cross-entropy, −ln p_true, the same objective as binary cross-entropy with K classes. For the row above with true class 1, the loss is −ln(0.659) ≈ 0.417; a model that had assigned 0.05 to the true class would pay 3.0, so the penalty for burying the truth still grows without bound. The gradient even keeps the famous clean form: dL/dz = p − y, a one-hot vector minus the predicted distribution. This is not a coincidence — binary logistic regression is the K = 2 special case of softmax: softmax of two logits equals the sigmoid of their difference. The two strategies you have now seen are one family, chosen by K and by the loss, not two unrelated models.

The behavioral difference is competition. In softmax, every class shares one denominator, so pushing one logit up steals probability from every other class; decision regions are carved out of a single pie and the boundary between classes k and j depends only on w_k − w_j. In OvR each class gets its own private sigmoid and never yields mass to anyone. That is why multinomial usually wins on mutually exclusive, reasonably balanced label sets — it is one coherent joint model — while OvR survives when labels overlap, when a class genuinely has nothing in common with the others, or when you need different features per class.

In scikit-learn, solvers lbfgs, newton-cg, and saga optimize the multinomial loss natively on multi-class data; liblinear only does one-vs-rest, and recent versions removed the multi_class knob entirely because multinomial is the default everywhere it applies. Check which one you are running before you trust the probabilities: an OvR-trained model and a multinomial-trained model on the same data give different predict_proba values and can disagree on the very rows near a boundary. The other practical consequence is regularization: the penalty now applies across the whole W matrix, so with many classes you may need a smaller C than a single binary problem required, and you should still scale features first — scaling does exactly what it did in the binary case.

Here is the multi-class failure you will be handed in an interview. A six-class product taxonomy hits 92% test accuracy, and the product manager is furious: not one item has been routed to garden tools in three weeks. Pull the per-class numbers and the story is immediate.

The accuracy is a support-weighted average, so it faithfully reports the four big classes and hides the two small ones. Both strategies you just learned produce this shape, through slightly different mechanics. Under OvR, the garden-tools classifier trained on 40 positives against 9,960 negatives — a 1:249 fight — and its 0.5 threshold never fires. Under softmax the minority class still loses: its decision region is squeezed because the dominant classes claim most of the feature space, and the loss term −ln p_true is only evaluated 40 times per epoch against thousands of majority terms.

Diagnose with three tools, in order. First, the per-class table above — precision and recall per class, not one global number. Second, the confusion matrix: garden tools is not being confused with sports, it is being absorbed into electronics, which tells you the features do not separate it, not just that the threshold is wrong. In a six-class problem the confusion matrix is 6×6 with true labels on rows and predictions on columns, and you read a minority row across: a row that is almost entirely off-diagonal, concentrated in one or two dominant columns, means the class was swallowed; a row smeared across sibling columns means the features are ambiguous. Third, the choice of aggregate metric. Macro-F1 averages the six per-class F1s with equal weight, so garden tools counts as much as electronics; weighted-F1 weights by support and reproduces the same blindness as accuracy. In single-label multi-class problems, micro-F1 pools every TP, FP, and FN globally and equals accuracy exactly — so if someone quotes a single number and it is micro or weighted, ask for the macro version. Macro-F1 is the default alarm bell for skewed multi-class data because it refuses to let volume drown out failure.

Fixing it is mostly the toolkit from the next topics: class weights rescale the loss so the 40 garden-tools rows matter as much as the 4,000 electronics rows, resampling synthesizes minority examples, and per-class thresholds re-balance the decision rule after training. One command reproduces the diagnosis on any model:

from sklearn.metrics import classification_report, f1_score

print(classification_report(y_test, pred, digits=3, zero_division=0))
print("macro-F1  ", f1_score(y_test, pred, average="macro"))
print("weighted-F1", f1_score(y_test, pred, average="weighted"))
# macro ~0.68 vs weighted ~0.92 on the table above: the gap IS the alarm

What to say when choosing a strategy outright: default to multinomial softmax when labels are mutually exclusive and roughly balanced; choose one-vs-rest when labels overlap, when "none of the above" is a real outcome, when per-class models need different feature pipelines, or when K is so large that one joint softmax is unwieldy. One mention of one-vs-one — K(K−1)/2 pairwise classifiers, the classic SVM multi-class scheme — rounds out the vocabulary; at six classes that is 15 models for what one softmax does, and it inherits none of softmax's probability semantics.

You now have models and scores; Confusion Matrix, Precision, and Recall turns those raw counts into the metrics you will actually quote in a design review.