Data Leakage Prevention

Define information available at prediction time

Data leakage occurs when a model uses information that would not be available at the moment a real prediction is made, or when evaluation data influences training. It produces an optimistic score that will not survive deployment. The central question is temporal and operational: for each feature, when is it created, when does it become available, and would the scoring system know it for the examples it must predict?

Target leakage is a feature that directly or indirectly encodes the outcome. A customer-status field updated after a churn event can reveal the label. A post-treatment measurement can contain the effect the model is supposed to predict. Such fields may be strongly predictive in a random split and useless or impossible at serving time. Build a feature availability table that names its source, timestamp, owner, and prediction-time availability.

Split according to the data-generating process

Random splitting is unsafe when future rows contain information related to earlier rows. For forecasting, train on earlier periods and validate on later periods. For repeated records from a person, patient, device, or account, keep groups together when deployment predicts for unseen entities. If the same entity occurs in both sets, the model can memorize identity-specific behavior and appear to generalize.

Preprocessing can leak too. Imputers, scalers, encoders, feature selection, and dimensionality reduction learn statistics from data. Fit them only on the training fold, then transform validation and test with the fitted object. A scikit-learn Pipeline or ColumnTransformer keeps these steps inside cross-validation. Target encoding must be computed out of fold or with an appropriate smoothing procedure; calculating category means from all labels leaks the target into validation features.

Watch time and aggregation boundaries

Aggregations can accidentally include future information. A rolling mean for a prediction at time t must use only records available before t. Group-level counts, averages, and target rates should be computed using the training window and a point-in-time join. Late-arriving events require an event-time and availability-time policy. A feature can be historically correct but still unavailable to the production system at decision time.

Label construction can leak too. If the label window overlaps a feature window, observations may contain consequences of the outcome. In fraud prediction, a chargeback status recorded weeks later cannot be an input for an earlier authorization decision. Document label horizon, feature cutoff, and any embargo period between training and validation when outcomes or windows overlap.

Detect suspicious results

Unusually high validation accuracy is a clue, not proof, of leakage. Compare against simple baselines, inspect feature importance and single-feature performance, and review data lineage for fields created after the target. Run a time-based or group-based split and test whether performance remains plausible. Validate the pipeline with a deliberately shuffled target; a model that still performs well may contain leakage or an evaluation bug.

Check for duplicate or near-duplicate rows across splits, entity overlap, and preprocessing fit scope. Recompute all derived features inside each training fold. Keep a holdout that reflects deployment and avoid using it repeatedly to choose features. If a suspected leak is removed and performance drops, investigate whether the earlier score depended on unavailable information before concluding that the model is worse.

Leakage prevention is a property of the complete data and evaluation workflow, not a final cleanup step. Point-in-time correctness, split design, preprocessing boundaries, and serving parity all need to be tested and documented.

Audit a churn dataset

Suppose a model predicts whether an account will churn in the next 30 days. The table includes plan type, recent usage, support contacts, cancellation-request date, and a field named account_closed. Before training, define the prediction timestamp and label window. account_closed may be written only after the customer leaves, so it is unavailable at prediction time and directly leaks the outcome. A cancellation request can also occur after the intended prediction date; its timestamp determines whether it is a valid feature.

Create a feature inventory with source, event time, availability time, transformation, owner, and expected serving path. Remove fields that are post-outcome or unavailable at scoring. For each rolling usage feature, ensure the aggregation ends before the prediction timestamp. Use point-in-time joins rather than joining current account state to historical training examples.

Build a leakage-safe evaluation

If the goal is predicting future churn for existing accounts, use a chronological split that trains on earlier windows and validates on later ones. If the goal is generalizing to new accounts, group by account and hold some accounts out entirely. A random row split can place the same account in both sets and inflate performance. If labels use a 30-day horizon, add a gap or embargo where overlapping observation windows would share information.

Place imputation, scaling, encoding, and feature selection inside a pipeline. During cross-validation, each fold must fit those steps only on its training portion. For target encoding, compute category statistics out of fold and apply smoothing; never use the validation labels to construct the encoded validation feature. Save the fitted pipeline and verify serving code uses the same column mapping and cutoff logic.

Test for leakage signals

Train a simple baseline and inspect very strong single-feature predictors. Ask whether each field exists at the prediction timestamp. Compare random-split performance with temporal and group-aware evaluation. If scores collapse, determine whether the random split was unrealistic, whether the population shifted, or whether a feature encoded identity or future events.

Shuffle training labels and rerun the full pipeline. Performance should fall toward the baseline. If it does not, inspect target-derived features, duplicate examples, join keys, and metric code. Search for the target or a transformed version in features and upstream tables. Check that feature selection and normalization are refit within each fold rather than once on the complete dataset.

Preserve evidence and ownership

Record the label definition, prediction timestamp, split strategy, feature cutoff, and preprocessing scope with the experiment. A reviewer should be able to trace one example and determine exactly which information was available when its prediction would have been made. Keep a test that fails if a known post-outcome field enters the feature set. Assign owners to upstream feature definitions and monitor availability delays in production.

When a feature is removed and validation performance falls, compare the new result with deployment-realistic data rather than restoring the leak. A lower but honest score can still support a useful model; a high score based on impossible information cannot. The goal is a trustworthy estimate of future performance, not the largest metric in an experiment report.

Leakage is not limited to target columns

The obvious case is a field that directly contains the label. More subtle leakage comes from time, identity, joins, preprocessing, human workflow, and repeated model selection. A feature may be legitimate for one prediction timestamp and invalid for another. Write the deployment moment first, then assess availability against it. This avoids treating a column's name as proof that it is safe.

Temporal leakage can occur through rolling aggregates, late events, label windows, or a random split across time. Group leakage occurs when related rows appear in both training and validation. Preprocessing leakage happens when learned transforms inspect holdout data. Target encoding and feature selection are especially easy to leak because the target is used in their construction. Put every learned step inside cross-validation and use point-in-time joins for historical features.

Treat surprising performance as a diagnostic

An exceptional score can reflect a genuinely easy task, but it should trigger data-lineage review. Inspect top features, compare simple baselines, evaluate temporal and group splits, and shuffle labels. Verify metric code and duplicate handling. If performance changes sharply after a realistic split, explain whether the original split tested a different task rather than silently selecting the more favorable result.

Do not remove a feature solely because it is highly predictive. Determine whether it is available and legitimate at the intended decision point. Conversely, do not retain a feature just because it improves cross-validation if production cannot compute it reliably. Availability latency and missingness are part of the model contract.

Interview framing

In an interview, define leakage as information unavailable at prediction time or information from evaluation data influencing training. Give a temporal example, a group-split example, and a preprocessing example. Explain point-in-time joins, pipeline-based cross-validation, and why a holdout should not be repeatedly tuned against. Then describe a diagnostic such as a label-shuffle test and a feature-lineage audit.

Finally, distinguish a lower honest score from a model failure. Evaluation is useful only when it approximates deployment. A leakage-safe pipeline, documented cutoffs, and serving parity are more valuable than a spectacular metric that cannot be reproduced in production.

Leakage review should include the human process around labels. A support agent may update a status after seeing the same outcome the model predicts, or an analyst may backfill a field using a future table. Ask the data owner how and when each feature is recorded, not only what its SQL column contains. Keep lineage and timestamp semantics available to reviewers.

There is a trade-off between strict point-in-time reconstruction and the engineering cost of reproducing historical availability. Prioritize features that materially affect the model and define a conservative fallback for uncertain fields. If historical snapshots are unavailable, do not assume current values were present at the original prediction time. Gather prospective data or restrict the evaluation claim to what can be supported.

When a leak is discovered after deployment, assess prior predictions and decisions, disable or recompute affected features, and notify the model owner. Preserve the investigation evidence and add a regression test so the field cannot re-enter unnoticed. Treat leakage as a data-governance issue as well as a modeling defect.