Feature Scaling and Normalization

A k-nearest-neighbors regressor trained on area_sqft (800–4,000) and rooms (1–6) will almost always find neighbors that match on area and ignore rooms entirely — not because rooms do not matter, but because the distance formula does not know that 3,000 sqft and 3 rooms are comparable amounts of variation. Squared Euclidean distance lets the large-range feature swamp the small one:

import numpy as np
from sklearn.neighbors import KNeighborsRegressor

X = np.array([[2000, 3], [2100, 4], [900, 2]])   # sqft, rooms
knn = KNeighborsRegressor(n_neighbors=1).fit(X, y)
knn.kneighbors([[2050, 3]])
# row 0 (2000 sqft, 3 rooms) wins: d^2 = 50^2 + 0^2 = 2500
# row 1 (2100 sqft, 4 rooms) loses: d^2 = 50^2 + 1^2 = 2501

A whole bedroom weighed less than 50 square feet. That is the failure scaling fixes.

The models that need scaling fall into three families, and the reason differs per family:

That last row is subtle and interview-worthy: L2 regularization squeezes every coefficient equally, so a feature measured in dollars (salary: 30,000–300,000) gets punished far harder than the identical signal measured in units of $100k (3–30). The penalty is not scale-invariant, so unscaled features silently receive unequal regularization — scaling makes the penalty fair.

Note also that scaling usually means two operations: centering (subtract the mean) and dividing by a spread. Some consumers only need one. PCA and neural-net initialization assume centered features; distance metrics need comparable spread; interpretable coefficients want both. StandardScaler does both, and knowing which half you need explains the with_mean=False setting you will meet in the sparse-matrix chapter.

Tree-based models need none of it. A split on sqft <= 1950 is a split regardless of units, and any monotone rescaling produces the identical tree. This is why a GradientBoosting pipeline without a scaler performs exactly the same with one — and why a candidate who reflexively scales everything has not understood the mechanism. One-hot outputs from Categorical Encoding are the classic features that must not be centered, for reasons the next chapters make concrete.

from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.svm import SVR

pipe = make_pipeline(StandardScaler(), SVR(kernel='rbf'))
pipe.fit(X_train, y_train)   # scaler learns from training rows only

Two practical notes to carry forward. First, fit the scaler on training data and transform validation and test with those same statistics — a scaler refit on validation hands the model the validation distribution, and the full mechanics of that mistake get their own treatment in Data Leakage Prevention. Second, scaling turns coefficients into standard-deviation units, which is a feature, not a side effect: after StandardScaler, comparing coefficient magnitudes across features becomes a legitimate importance sanity check.

StandardScaler is the right default, but three common distributions pull you toward the alternatives, and the choice is decided by two questions: does the column carry heavy outliers, and does the model care about the range or only the shape?

Code, side by side, on a column that is 95% under $200 with a few $10,000 tails:

import pandas as pd
from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler

x = pd.DataFrame({'spend': [12, 18, 25, 150, 9800, 20, 14]})
for scaler in (StandardScaler(), MinMaxScaler(), RobustScaler()):
    out = scaler.fit_transform(x)[:5, 0]
    print(type(scaler).__name__, out.round(2))
# StandardScaler  [-0.32 -0.32 -0.32 -0.22  3.08]
# MinMaxScaler    [ 0.    0.    0.    0.01  1.  ]
# RobustScaler    [ 0.05 -0.05  0.09  0.98 71.3 ]

Read those outputs carefully — the same column tells three stories. StandardScaler's variance is essentially the outlier's doing, so every ordinary row collapses toward −0.32 and the model can barely distinguish a $12 customer from a $25 one. MinMax is worse: the entire normal population occupies the bottom 1% of [0, 1]. RobustScaler keeps the middle spread intact and lets the outlier stand out — which is what you want when the outlier is real signal you refuse to delete.

But for this column the best scaler may be no scaler at all — a log1p transform first:

import numpy as np
df['log_spend'] = np.log1p(df['spend'])
# [2.56 2.94 3.26 5.02 9.19 3.04 2.71]  <- spread restored, outlier defanged

Heavy-tailed money and count columns are usually log problems before they are scaling problems, and log-then-StandardScaler is the standard production recipe for them. A useful heuristic: if the mean is more than about three times the median, reach for the log before any scaler.

Ordering rules that survive code review:

1. Fit scalers inside the training fold, never before the split — fit_transform on train, transform on validation, or let a Pipeline own both calls.
2. Preserve sparsity: on one-hot outputs use StandardScaler(with_mean=False); centering a sparse matrix densifies it, and a 200k-column matrix eats the machine. (MaxAbsScaler is the sparse-safe alternative when you want range control without density.)
3. MinMax on data with a known physical bound (images, rates in [0, 1]) is fine; on unbounded data, test rows escape the [0, 1] range — harmless for trees, surprising for neural nets.
4. Scaling is per-feature. Row normalization (sklearn.preprocessing.normalize) is a different operation, used for row vectors like TF-IDF documents — do not confuse the two in an interview.

The third and final trap is where scaling implementations go wrong in practice: sparse inputs that explode memory, and scalers fitted one line too early in the script.

A correct scaler in the wrong place still breaks the model, and the two ways it breaks — memory and leakage — share one fix: put the scaler inside the Pipeline.

Sparsity first. One-hot and TF-IDF features arrive as scipy.sparse matrices with 99%+ zeros. StandardScaler defaults to with_mean=True, which builds a dense mean vector and subtracts it from every element — sparse becomes dense instantly:

from scipy.sparse import random as sparse_random
from sklearn.preprocessing import StandardScaler

S = sparse_random(100_000, 200_000, density=0.001, format='csr')  # 20M stored values
StandardScaler().fit_transform(S)                  # MemoryError on many machines
StandardScaler(with_mean=False).fit_transform(S)   # stays sparse; per-column variance only

with_mean=False keeps zeros as zeros, turning the transform into a pure per-column rescale — exactly what a bag-of-words column needs. The tell during code review: a pipeline that OOMs the moment the encoder is added usually forgot this flag. (Variance scaling still works without centering; if a model truly needs centered sparse features, the honest answer is to densify a reduced matrix, not the full one.)

Leakage second. Scaling statistics are learned quantities, and this code is wrong in a way that survives casual inspection:

Xs = StandardScaler().fit_transform(X)       # learns mu, sigma from ALL rows
score = cross_val_score(model, Xs, y, cv=5)  # folds scored on statistics they helped produce

Each validation fold contributed its rows to μ and σ — a small leak, often worth a point of AUC, invisible except as a persistent gap between cross-validation and production. The fix is structural rather than disciplined:

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
cross_val_score(pipe, X, y, cv=5)   # scaler refits inside every fold

cross_val_score clones the pipeline per fold, calls fit on the training portion only, and scores with statistics the validation rows never touched. You cannot keep this right by being careful across a dozen experiments; you get it right by making the wrong arrangement impossible to express — the same guarantee that Categorical Encoding targets when encoders are moved into pipelines.

Verification habits. After fit on a training fold, means should sit near 0 and variances near 1 for standard-scaled columns; the fitted objects hand you the numbers directly:

from sklearn import set_config
set_config(transform_output='pandas')   # every step emits DataFrames while debugging
scaled = pipe[:-1].fit_transform(X_train)
scaled.mean().abs().max()               # ~1e-17 for standard-scaled numeric columns
pipe.named_steps['standardscaler'].mean_[:5]

One last temptation to reject: scaling the target. For regression, standardizing y changes the objective's units, so your MAE now reads in standard deviations instead of dollars and every stakeholder conversation needs a translation step. Scale y only when the estimator itself demands it, and invert the transform on predictions before reporting.

With every feature finally on comparable scales, you can build new ones that exploit the comparisons — that is Feature Engineering Recipes, the next topic, where logs, date parts, and interactions turn the cleaned columns into actual signal.

A common scikit-learn pattern is to put the scaler and estimator in one Pipeline, then pass that pipeline to cross-validation or a search object. Each training fold fits its own scaler; the corresponding validation fold only receives transform. For a final holdout, call fit on the training partition and evaluate with predict on the untouched holdout. Calling fit_transform on the whole table before splitting leaks information even when labels are not used, because the holdout's distribution influenced the means, variances, minima, or quantiles. Persist the complete fitted pipeline so inference applies exactly the learned transformation. After deployment, monitor feature ranges and missingness: new values far outside training ranges can signal drift or a broken upstream unit conversion.