Logistic Regression Fundamentals

An interviewer writes a score on the whiteboard and opens with the classic cold-start: what does logistic regression actually model? The answer that loses the room is "the probability of class 1." The answer that wins names the quantity precisely: logistic regression models the log-odds of the positive class as a linear function of the features. The sigmoid, the 0.5 cutoff, and the cross-entropy loss are all consequences of that one commitment.

Start with a probability p between 0 and 1. Its odds are p / (1 − p): odds of 4 means four chances to one, so p = 0.8; odds of 0.25 means p = 0.2. Odds are always positive and unbounded above, which is still awkward for a linear model. Take the logarithm and you get the log-odds, also called the logit: log(p / (1 − p)). Now the range is the entire real line — p = 0.5 maps to 0, p → 1 maps to +∞, p → 0 maps to −∞ — and the map is symmetric: doubling the odds of a coin flip moves you from 0 to +0.693, halving moves you to −0.693.

So logistic regression fits weights w such that wᵀx estimates log(p / (1 − p)) — a straight line in log-odds space, where least-squares machinery works comfortably. To return to a probability you invert the logit, and that inverse is the sigmoid:

p = σ(z) = 1 / (1 + e^(−z)), where z = w₀ + w₁x₁ + … + wₙxₙ

The identity σ(logit(p)) = p is the entire design: the linear part lives where the algebra is easy, and the sigmoid translates its output back into probability space.

import numpy as np

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

w = np.array([-2.0, 3.0])   # intercept, weight on "failed_payments"
x = np.array([1.0, 1.0])    # one failed payment
z = w @ x                   # -2 + 3 = 1.0
p = sigmoid(z)              # 0.731 -> P(default) = 73%
print(f"{z=}, {p=}")

One more reading of the pieces: the intercept w₀ is the log-odds when every feature is zero, which is a meaningful baseline only when features are centered — a small reason scaling belongs in the pipeline. And the sigmoid is not a claim that nature is S-shaped; it is a link function chosen because it is smooth, invertible, and maps a linear predictor onto a valid probability, which is what keeps the likelihood well behaved while the optimizer moves w around.

Because the sigmoid is strictly increasing, σ(z) ≥ 0.5 exactly when z ≥ 0. The decision boundary is therefore wᵀx = 0 — a hyperplane in feature space no matter how curved the probability curve looks. That is the model's structural claim: it bends probability smoothly, but the line where it flips its answer stays linear. Feed it features whose classes are not linearly separable and it will plateau at whatever accuracy that line allows, which is why feature transforms show up next to logistic regression so often.

The log-odds commitment also hands you the interpretation interviewers ask for immediately after. Each coefficient is a log-odds slope, so e^w is an odds ratio: with w₁ = 3, one extra failed payment multiplies the odds of default by e³ ≈ 20.1, holding every other feature fixed. Contrast linear regression, where a coefficient adds units to the target — here a coefficient multiplies odds, and it only means anything one feature at a time with the rest pinned.

Notice the shape of what you just built: one linear score squashed into one probability. With K classes you either run that recipe K times or normalize several scores against each other — precisely the fork that Multi-class Strategies dissects. First, though, the objective function: why training a probability model never uses squared error.

You can fit w by gradient descent, so you must choose a loss. The tempting shortcut — reuse linear regression's mean squared error on 0/1 labels — is wrong twice over: its gradient nearly vanishes exactly when the model is most wrong, and its error surface is non-convex in w. Logistic regression trains on cross-entropy (log loss) instead, and the reason shows up in one example's arithmetic.

For a single example with label y ∈ {0, 1} and predicted probability p, the loss is:

L = −[ y·ln(p) + (1 − y)·ln(1 − p) ]

When y = 1 this is just −ln(p): p = 0.9 costs 0.105, p = 0.5 costs 0.693, p = 0.1 costs 2.303, p = 0.01 costs 4.605. The penalty for being confidently wrong grows without bound — ln(0) → −∞ — so the loss keeps pushing hard on the very examples the model misses. Squared error caps its penalty at 1 and, as the gradients below show, barely pushes at all.

The gradient is where the two losses genuinely diverge. For log loss on top of a sigmoid, ∂L/∂z = (p − y), so the update on weight j is (p − y)·x_j: large whenever the prediction is wrong, zero only when p matches y. For squared error the chain rule drags in the sigmoid's own derivative: ∂L/∂w = (p − y)·σ′(z)·x, and σ′(z) = σ(z)(1 − σ(z)) peaks at 0.25 and collapses toward 0 as |z| grows. Take z = −6 with y = 1 (the model says p ≈ 0.0025, but the truth is 1). The log-loss gradient is about −0.9975; the squared-error gradient is about −0.9975 × 0.00246 ≈ −0.0025 — a 400× weaker push on the most badly wrong prediction in the batch. Cross-entropy keeps its gradient because it never multiplies by σ′.

import numpy as np
from sklearn.metrics import log_loss

y = np.array([1, 0, 1])
p = np.array([0.9, 0.8, 0.2])

eps = 1e-15
bce = -np.mean(y * np.log(p + eps) + (1 - y) * np.log(1 - p + eps))
print(bce)          # 1.108
print(log_loss(y, p))  # 1.108 — sklearn clips p to [1e-15, 1-1e-15] so log(0) never happens

A related interview question: why not train directly on accuracy? Because accuracy is a step function of the scores — nudging a probability from 0.49 to 0.51 flips a prediction, yet the accuracy gradient is zero almost everywhere, so gradient descent has nothing to follow. Cross-entropy is the smooth surrogate with the same minimizer: pushing each prediction toward its label is the differentiable version of counting more correct predictions.

There is a statistical story on top of the optimization one. Minimizing average log loss over a dataset is exactly maximum likelihood estimation for a Bernoulli model whose parameters are the predicted probabilities — the same objective, written as a loss. And log loss is a strictly proper scoring rule: expected loss is minimized only when p equals the true conditional probability, so a model that answers 0.7 when reality is 0.7 pays less than any other answer. Training objective and deployment question ("are these probabilities calibrated?") become the same question.

Log loss is also the term every later remedy plugs into: multiply each example's term by a class weight and the optimizer starts caring about the expensive class, or leave training alone and sweep the probability cutoff at serving time. Before either, the practical failure modes — what actually breaks when you fit this model on real features.

Your fit finishes with coefficients of ±47, scikit-learn prints ConvergenceWarning: lbfgs failed to converge, and every predicted probability sits at 0.000 or 1.000. Inspect the data and you find a feature that separates the classes perfectly — chargebacks == 0 on every clean row, ≥ 1 on every fraudulent one. This is complete separation: the likelihood keeps improving as the weights grow, so no finite maximum-likelihood estimate exists and the optimizer chases w toward ±∞. It is a property of the data, not a bug in your code.

The standard cure is the one you already met in Ridge, Lasso, and Elastic Net: penalize the weights. scikit-learn's default LogisticRegression(C=1.0, penalty='l2') minimizes log loss plus a term proportional to ‖w‖² / (2C) — the exact constant depends on the solver — which makes the objective finite again: the data wants the weights large, the penalty wants them small, and the compromise lands somewhere usable. Remember C is the inverse of strength, the support-vector-machine convention rather than alpha's: halve C and you double the shrinkage. If you want the zeroed-out coefficients that Lasso gave you, penalty='l1' with solver='liblinear' or 'saga' doubles as feature selection, and penalty='elasticnet' with an l1_ratio gives the middle road.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

model = make_pipeline(
    StandardScaler(),
    LogisticRegression(C=0.1, penalty="l2", class_weight="balanced",
                       solver="lbfgs", max_iter=1000),
)
model.fit(X_train, y_train)
model[-1].n_iter_[0]        # healthy fits converge in far fewer than max_iter

Second failure mode: features on wildly different scales. annual_income in dollars (30,000–300,000) beside num_children (0–5) makes the loss surface long and thin — gradient steps bounce across the narrow dimension, n_iter_ climbs toward max_iter, and convergence warnings appear on healthy data. The fix is mechanical: put StandardScaler inside a Pipeline so scaling is learned on training folds only. In practice, read n_iter_ and ConvergenceWarning as a triage signal — scale first, raise max_iter, and only then suspect the model.

Third: correlated features. Logistic regression spreads credit across them arbitrarily, so two proxies for the same signal can trade signs and you end up claiming the thing that protects users is what harms them. Predictions stay stable; interpretation does not. If coefficients are the deliverable — and in an interview they always are — drop or combine correlated features before fitting, or state the instability outright.

Fourth, know the assumptions you are signing up for: independent rows, a linear log-odds relationship, and no interactions unless you build them. If risk actually rises for both very young and very old drivers, a plain model on raw age averages the two tails into a flat, misleading coefficient. You do not abandon the model — you add the term that lets the log-odds bend (age, age², or an interaction), and then interpret coefficients on those transformed features instead.

That leads to the question following every logistic regression answer: interpret this coefficient. For feature j, a one-unit increase multiplies the odds of the positive class by e^{w_j} holding the others fixed; after standardization the statement becomes "per standard deviation," which is why pipelines make coefficients comparable. For w_j = −0.69, e^(−0.69) ≈ 0.50: the odds halve. Give the direction, the odds ratio, and what is held constant, and you have answered it.

Everything so far trained one score against one label. Real problems hand you ten product categories, three severities, four fault types — the sigmoid's single probability has to become a full distribution over classes, which is the decision Multi-class Strategies works through next.