Missing Values and Outliers

Your dataset reports 12% missing values in annual_income and you must decide what to do before training. The wrong first move is reaching for an imputer. The right first move is asking a diagnostic question: why is the value missing? The answer changes your strategy and how much you can trust the final model, and interviewers know it.

Statisticians name three mechanisms, and every real dataset is a mixture of at least two of them.

The distinction matters because each mechanism implies a different bias. Under MCAR, dropping rows loses nothing but statistical power. Under MAR, filling the whole column with its global mean is wrong because the missing subset has a different mean than the full column — you must impute conditionally. Under MNAR, the act of being missing is a feature: the customers who skip the income field are systematically different, and no imputer can reconstruct what they refused to tell you — but a binary indicator column can exploit the refusal.

You cannot prove the mechanism from data alone — that requires domain knowledge about the collection process — but you can gather strong evidence. The cheapest test is to compare other columns between rows where annual_income is missing and rows where it is present.

import pandas as pd

df = pd.read_csv('applications.csv')
mask = df['annual_income'].isna()

print(df.groupby('employment_status')['annual_income']
        .apply(lambda s: s.isna().mean()))
# employment_status
# employed     0.04
# student      0.31
# retired      0.28

A 4% gap rate among employed rows versus 31% among students is a red flag for MAR: missingness depends on an observed column. If instead the missing rows look identical on every observed column yet their reported incomes (when present) cluster at the extremes, you are facing MNAR, and you should say so in your write-up rather than silently imputing.

Two practical habits follow from the diagnosis. First, build missingness indicators for any column whose gap rate exceeds a few percent and whose mechanism is not clearly MCAR:

df['income_missing'] = df['annual_income'].isna().astype(int)

Second, remember that most scikit-learn estimators refuse NaN outright (trees excepted), so the mechanism diagnosis happens before you write the preprocessing code — the mechanism decides whether you impute conditionally, add indicators, or both. The next chapter compares the actual imputation tools and the bias-versus-sample-size trade each one makes.

Having established that the gap itself can carry information, you already own the statistical vocabulary to defend the group comparison above: in Hypothesis Tests and Confidence Intervals you learned to ask whether a difference between two groups is real or noise — the same question, applied here to the missing and non-missing subsets.

Once you know roughly why values are missing, you pick a strategy — and every strategy pays in one currency and charges you in another. Deleting rows buys statistical purity and charges sample size. Mean imputation buys sample size and charges you understated variance and, under MAR, bias. Model-based imputation buys both and charges you complexity plus the risk of smuggling target information into features.

The median-versus-mean choice is not about taste. Under skewed distributions the mean is dragged toward the tail, so filling every gap with the mean systematically misrepresents the majority: in a column of incomes with a few millionaires, the mean describes nobody. The median is robust to that. And regardless of which statistic you pick, fit the imputer on the training fold only — a median computed over train plus test has already seen the validation distribution.

from sklearn.impute import SimpleImputer, KNNImputer, IterativeImputer

simple = SimpleImputer(strategy='median')                  # fast, stable
knn    = KNNImputer(n_neighbors=5)                        # borrows from similar rows
iter_  = IterativeImputer(max_iter=10, random_state=0)    # chained regressions

X_train_imp = iter_.fit_transform(X_train)   # learns column relationships
X_test_imp  = iter_.transform(X_test)        # reuses them, never refits

IterativeImputer fits one regression per feature on the other features, fills the gaps, and cycles until convergence. It performs well under MAR — but notice what it needs: every other column it leans on must be available at both training and serving time. If one of those columns is itself missing in production rows, you have built a scorer that cannot score.

KNNImputer deserves its own caveat: it averages the k nearest observed rows, which means it needs scaled inputs to define nearness, and it inherits the curse of dimensionality — with 60 features, nearest neighbors stop being neighbors. Use it when the feature count is modest and row-level similarity is genuinely meaningful.

A decision procedure that survives code review:

1. Under 1–2% missing and plausibly MCAR: drop the rows and note the loss in your summary.
2. Under MAR with an obvious conditioning column: impute within groups, and keep the indicator.
3. Heavy tails or many columns missing: median plus indicator, then check whether the model uses the indicator — it often does, and that is the MNAR signal from the previous chapter surviving into the features.
4. Rich correlations and a modest feature count: try IterativeImputer inside cross-validation and keep it only if fold scores improve, not the in-sample score.

Whichever branch you take, the imputer belongs inside the cross-validation loop. An imputer fitted before the split contaminates every fold with statistics from rows it should never have seen — the exact failure mode that gets a full treatment in Data Leakage Prevention.

A single row with transaction_amount = 9_500_000 in a dataset of $30 purchases will not merely hurt your model — it will define it. Squared-error losses square that error too; gradient-based optimizers take their largest steps on exactly those rows; and any scaler with finite variance lets the outlier dictate the range everything else gets squashed into. Outlier handling is not cosmetics, it is loss-function hygiene.

Two detector families cover most interview scenarios: geometric rules and isolation-based tree ensembles.

IQR fences come from the five-number summary and assume nothing about the distribution. With Q1 and Q3 as the first and third quartiles:

import pandas as pd

q1, q3 = df['transaction_amount'].quantile([0.25, 0.75])
iqr = q3 - q1
lo, hi = q1 - 1.5 * iqr, q3 + 1.5 * iqr      # 'moderate' outliers
lo2, hi2 = q1 - 3 * iqr, q3 + 3 * iqr        # 'extreme' outliers

flagged = ~df['transaction_amount'].between(lo2, hi2)

The 1.5x and 3x multipliers are Tukey's conventions, not laws — widen them for skewed data, or apply the fences to log-transformed values, which is often the better fix anyway. IQR is univariate, deterministic, and explainable to a compliance reviewer, which keeps it the default for reporting.

IsolationForest handles the multivariate case: an outlier is easy to isolate, so random splits isolate anomalies in few steps, and the average split depth over the ensemble becomes an anomaly score.

from sklearn.ensemble import IsolationForest

iso = IsolationForest(n_estimators=300, contamination=0.02, random_state=0)
df['anomaly'] = iso.fit_predict(df[['amount', 'duration', 'device_count']])
# -1 marks outliers, 1 marks inliers

contamination sets the expected outlier fraction and therefore the decision threshold — set it from a business prior (about 2% of transactions are fraud) rather than tuning it to flatter a score. IsolationForest also catches the point that is individually normal on every feature but absurd in combination: a $900 purchase on a card created ten minutes ago.

Detection done, you choose a response, and the response depends on whether the extreme value is an error or a real event:

Capping in practice:

hi = df['transaction_amount'].quantile(0.99)
df['transaction_amount'] = df['transaction_amount'].clip(upper=hi)

Note the fitting order one more time, because it is the theme of this entire phase: the quantiles, the fences, and the IsolationForest are all learned statistics. Compute them on the full dataset and your validation fold has leaked into preprocessing. Fit on the training fold, apply the learned bounds to validation, and the recipe holds. The next topic, Categorical Encoding, turns raw category columns into numeric features — and the same fit-on-train-only discipline applies to every encoder you will meet there.