Categorical Encoding

OneHotEncoder is the default answer to 'how do I encode a category column', and most of the time it is the right one. The skill interviewers are probing is whether you know its failure boundary: one-hot pays a fixed price per category, and some columns cannot afford it.

For a column with k distinct categories, one-hot produces k binary columns. Each is a 0/1 indicator, so no ordering is implied — city_London and city_Paris are unrelated coordinates, and a linear model learns an independent weight per category. That independence is the encoding's honesty: it never invents an order that does not exist.

from sklearn.preprocessing import OneHotEncoder

ohe = OneHotEncoder(handle_unknown='ignore', sparse_output=False)
X_enc = ohe.fit_transform(df[['color', 'city']])
ohe.get_feature_names_out(['color', 'city'])
# ['color_blue', 'color_green', 'city_London', 'city_Paris', ...]

Three parameters decide whether one-hot survives contact with production:

handle_unknown is the setting people skip. Leave it at the default and your pipeline trains perfectly on Tuesday, then crashes on Thursday when a user types a city the training data never saw. 'ignore' degrades gracefully — the unseen category looks like the dropped baseline — but remember that graceful degradation is not information: a model that one-hot encoded city has no mechanism to generalize across cities, which is precisely why high-cardinality columns need a different tool.

The budget rule: one-hot is safe while k is small relative to your row count — think tens of levels (color, device type, plan tier). It gets uncomfortable in the hundreds (zip codes) and unusable in the tens of thousands (user IDs, product SKUs, token streams), where a 200,000-column matrix of 10,000 rows means 2 billion floats dense, and even sparse, a regularized linear model must estimate 200,000 coefficients from 10,000 samples — most of them chasing noise. The encoder's own max_categories parameter exists to collapse the long tail of rare levels before that happens.

Note what one-hot is not: an answer for ordered categories. Encoding size = {XS, S, M, L, XL} as one-hot throws away ordering you already know, forcing the model to relearn S < M < L from data a child could state. That case belongs to ordinal encoding, and the trap inside it is the next chapter.

All of this is still the first half of the story: one-hot and ordinal both collapse when the column is an identifier — 40,000 merchants, 200,000 SKUs — and the escape routes (target and frequency encoding) carry a leakage tax you must know before you use them. The mechanics of dealing with the fallout from imputers and encoders fitting on too much data live in Missing Values and Outliers; what follows is how to encode the columns they left numeric-but-unordered.

Ordinal encoding maps each category to a single integer, which is exactly right when the integers mean something and exactly wrong when they do not. The whole topic reduces to one question per column: does the column have a real order, and is the spacing honest?

from sklearn.preprocessing import OrdinalEncoder

sizes = ['XS', 'S', 'M', 'L', 'XL']
oe = OrdinalEncoder(categories=[sizes])   # order declared, not inferred
df['size_code'] = oe.fit_transform(df[['size']])

Passing categories explicitly is not ceremony. The encoder otherwise sorts alphabetically, which happens to work for XS–XL here but silently turns ['Small', 'Medium', 'Large', 'XL'] into Large < Medium < Small < XL. Declare the order, or do not use ordinal encoding at all.

Why does an invented order hurt? It depends on the model, and this distinction is a favorite interview probe:

• Tree models (RandomForest, XGBoost) split on code <= 2. Any monotone relabeling yields identical splits, so an arbitrary integer code costs them nothing — a tree can chop the invented order into as many pieces as it needs.
• Distance and linear models see the numbers whole. For kNN or an SVM, city_code = 7 is 3.5 times city_code = 2, and switching between one-hot and ordinal changes who each row's neighbors are. For linear regression, the integer code imposes a straight line through the levels: it asserts that going S to M has the same effect as M to L, which may be plainly false (an XL shirt uses half again the fabric of an L).

When the order exists but the spacing does not, you have three honest options: keep the integer code and accept the linearity assumption, one-hot the levels and let the model learn per-level weights, or use a scoring dictionary that encodes real spacing ({'XS': 0, 'S': 1, 'M': 2, 'L': 3, 'XL': 5} if XL really jumps). State which one you chose and why — that one sentence is most of the answer to the interview question.

The trap examiners like to catch is LabelEncoder on a feature column. It was designed for the target variable in classification; applied to features it silently imposes alphabetical order, returns a single column with no handle_unknown story, and cannot transform a validation set whose categories differ. OrdinalEncoder for features, LabelEncoder for the target only.

Integer codes also interact with the cleaning work from Missing Values and Outliers: OrdinalEncoder can reserve a code for np.nan via encoded_missing_value=, preserving missing-as-signal instead of crashing. For columns with no order and thousands of levels, though, neither one-hot nor ordinal will do — that is the job of target and frequency encoding, with all the leakage risk they drag along.

A merchant_id column with 40,000 levels breaks one-hot, and ordinal encoding would tell your SVM that merchant 800 is 'twice' merchant 40. The standard escape is target encoding: replace each category with the mean of the target among rows carrying that category. One column per feature, and the numbers carry real signal — but the signal is exactly what makes it dangerous.

# the naive version, and why it is wrong
mapping = y.groupby(df['merchant_id']).mean()   # computed on ALL rows
df['merchant_te'] = df['merchant_id'].map(mapping)

The mistake in those two lines is the whole lesson: every row's feature now contains a statistic that includes that row's own label. A merchant with three fraudulent transactions out of three gets an encoded value of 1.0 on each of its rows — the feature is the answer. Training error collapses, cross-validated scores look spectacular, and the model dies in production, where no future label exists to compute the statistic from. This is target leakage in its most common interview form.

The safe version computes each row's encoding from other rows — out-of-fold — then blends toward the global mean so rare categories do not encode pure noise:

import numpy as np
from sklearn.model_selection import KFold

def target_encode(train, col, y, n_splits=5, smoothing=20):
    global_mean = y.mean()
    oof = np.zeros(len(train))
    kf = KFold(n_splits, shuffle=True, random_state=0)
    for tr_idx, val_idx in kf.split(train):
        means = y.iloc[tr_idx].groupby(train[col].iloc[tr_idx]).mean()
        counts = train[col].iloc[tr_idx].value_counts()
        shrunk = (means * counts + global_mean * smoothing) / (counts + smoothing)
        oof[val_idx] = train[col].iloc[val_idx].map(shrunk).fillna(global_mean)
    return oof

smoothing blends each category's mean toward the global mean in proportion to its support: a category seen three times leans almost entirely on the global mean; one seen 3,000 times keeps its own. Fitting the mapping out of fold is what separates a feature from a leak — each row's encoded value is computed from labels it did not contribute to.

If out-of-fold machinery feels heavy, leak-free alternatives exist:

Frequency encoding is the safe default when you want some signal from a 40,000-level column and cannot risk leakage: a rare SKU is rare at training and at serving, consistently. Target encoding earns its risk premium when category identity genuinely correlates with the outcome (some merchants really are fraud magnets), which is why production versions wrap it in folds, smoothing, and a fallback to the global mean for unseen categories.

Whatever you choose, build the encoded column inside the pipeline that trains the model, so cross-validation refits the mapping per fold. With the category columns finally numeric, the next topic, Feature Scaling and Normalization, is where those encoded columns and the raw numerics beside them get their scales under control — and one-hot outputs are the exact case where scaling must not center.