Regression Evaluation Metrics

Choose a metric that reflects error cost

Regression metrics summarize prediction error, but each weights mistakes differently. Mean absolute error (MAE) averages absolute residuals and remains in target units. It is relatively robust to a small number of very large errors compared with squared-error metrics. Mean squared error (MSE) squares residuals, so large misses dominate. Root mean squared error (RMSE) takes the square root of MSE and returns to the target's units while retaining that larger-error emphasis.

Choose based on the consequence of errors. If a few severe misses are especially costly, RMSE may highlight them. If the goal is a typical absolute miss, MAE may be easier to explain. A metric does not encode all business costs automatically. If overprediction and underprediction have different consequences, consider asymmetric loss or quantile regression and evaluate the relevant decision outcomes.

Interpret R-squared carefully

R-squared compares residual variation with variation around the mean baseline on the evaluated data. It can be interpreted as the fraction of variance explained in-sample under the standard OLS setup, but it does not say predictions are accurate in target units. A model can have a high R-squared and still make errors too large for the application. On held-out data, R-squared can be negative when predictions are worse than the evaluation-set mean baseline.

Adding predictors cannot reduce ordinary in-sample R-squared, even when a feature contributes little. Adjusted R-squared penalizes model size under its assumptions, but it is not a substitute for a deployment-aligned validation split. Do not compare R-squared across targets or populations as if it were a universal quality score. Inspect error magnitude, residual structure, subgroup performance, and baseline comparison.

Understand percentage metrics

Mean absolute percentage error divides errors by actual target values. It is easy to communicate as a percentage but behaves badly when actual values are zero or near zero, where the denominator makes the metric undefined or enormous. It also weights the same absolute error more heavily for small targets, which may or may not reflect business cost. Symmetric MAPE changes the formula but still has interpretation and edge-case limitations.

If percentage error is meaningful for the domain, define handling for zeros and explain weighting. Alternatives include MAE by segment, normalized error relative to a stable scale, or a business-specific cost function. Never report only one metric when its edge cases could conceal failure. A metric dashboard should show sample counts and data coverage alongside the score.

Use residual diagnostics and honest splits

Residuals are observed minus predicted values. Plot them against predictions, features, time, and important groups. A curve may indicate missing nonlinear structure; a funnel can indicate changing variance; temporal patterns may indicate drift or correlated errors. Metrics aggregate those patterns into a number, so diagnostics reveal where the model fails.

Fit preprocessing and model selection only on training data. Use a validation set or cross-validation for tuning, then evaluate once on an untouched test set. For future predictions, split by time; for new entities, split by entity group. A random row split can leak related observations and overstate performance. Compare against a simple baseline and include uncertainty when scores vary across folds or periods.

Compare two prediction models

Suppose a demand model has held-out residuals of 2, 3, 4, and 20 units. MAE is 7.25 units, while RMSE is larger because the 20-unit miss receives more weight. A second model with residuals of 6, 7, 7, and 7 has MAE of 6.75 and a lower RMSE as well. The first model is slightly worse on the typical absolute miss and much worse on the large-error sensitivity. Whether the second is preferable still depends on under- versus overprediction costs and the target range.

Calculate the same metrics on a naive baseline, such as predicting a seasonal average or the training mean. If the model only marginally improves on a robust baseline, its operational complexity may not be justified. Report the evaluation period, sample count, target units, and split strategy. Scores without those details cannot be compared reliably.

Inspect where errors occur

Plot residuals against predicted demand and time. If errors grow with demand, the model may be underestimating peaks or showing heteroskedasticity. Compute MAE by season or product group to identify uneven performance. Check calibration by grouping predictions into bins and comparing average prediction with average outcome. A low global score can hide systematic bias in a group that matters to the business.

Try MAPE only if actual values stay safely away from zero and relative error reflects the decision cost. For products with low-volume periods, MAPE can dominate the score with near-zero denominators. Define how missing or zero targets are treated and show counts excluded. Consider a weighted or asymmetric metric if the cost of a stockout differs from the cost of overstock.

Preserve a clean evaluation boundary

Use cross-validation within training data for model selection. Keep a final test set untouched until choices are fixed. If multiple configurations are repeatedly evaluated on the same test set, the test set becomes part of the tuning loop. For time-dependent demand, train on earlier periods and validate on later periods. Consider rolling-origin evaluation when performance across several future windows matters.

Preprocessing is part of the model. Fit imputers, scalers, encoders, and feature selection only inside each training fold through a pipeline. Leakage can make every metric look better while production performance disappoints. Record the feature schema and training cutoff so the evaluation can be reproduced.

Communicate results for a decision

Report one primary metric tied to the decision, supporting metrics that expose tails, and diagnostics for important groups. State the acceptable error threshold and how often it is exceeded. Compare against baseline, describe uncertainty across folds or periods, and identify one failure mode. A model is not good because its metric improved; it is useful when the improvement is robust and changes a decision without introducing unacceptable risk.

Check metric calculations against a small hand-computed example before adopting a library report. Confirm whether the library averages per row, weights by sample, or aggregates by group. A weighted metric can be appropriate when exposure differs, but the weighting rule must match the product decision and be documented. Keep a consistent evaluation population so apparent improvement is not caused by changing which records were included.

No single score captures the whole model

MAE, MSE, and RMSE use different error weighting. R-squared is relative to a mean baseline and can be negative on held-out data. Percentage metrics can be unstable near zero. The metric should reflect the decision, not merely the one that produces the most attractive number. If overestimation and underestimation have unequal costs, symmetric metrics may mislead.

Aggregate metrics can hide subgroups, time periods, and extremes. Report segment performance when errors affect populations differently. Show sample counts and confidence or fold variation. A narrow average improvement may not justify additional system complexity if the metric changes substantially by month or region.

Protect against optimistic evaluation

Random splits can leak repeated entities, future information, or preprocessing statistics. Match the split to the actual deployment: temporal split for future periods, group split for unseen entities, and nested validation when model selection is extensive. Keep the final test data separate. Fit transformations within each training fold and preserve the cutoff and schema.

Residual plots complement metrics. They reveal nonlinear patterns, changing variance, outliers, and drift. A model can have low RMSE while failing systematically at the high end. Review prediction intervals or uncertainty estimates when consequences vary with confidence, and monitor production metrics after deployment.

In an interview, define each metric's units and error weighting, explain the danger of MAPE near zero, and distinguish R-squared from absolute accuracy. Describe a baseline, a deployment-aligned split, and one residual diagnostic. Tie the metric to the cost of a wrong prediction. This shows evaluation is a design decision rather than a scoreboard.

Metric choice affects model selection. If a forecast supports inventory purchasing, underprediction may create stockouts while overprediction creates holding cost; a symmetric absolute error does not express that asymmetry. Quantile loss can estimate a chosen conditional quantile, and a cost function can reflect downstream consequences. These approaches require a clear decision target and calibration checks; selecting a metric after seeing which model wins invites bias.

Uncertainty in the reported score matters. Cross-validation fold variation, bootstrap intervals, or performance across several future windows can show whether an improvement is stable. Do not treat a tiny average gain as meaningful when it is smaller than evaluation noise or creates a worse tail outcome. Report the observation window, sampling unit, and number of examples so readers can judge coverage.

Subgroup analysis should be planned with care. Small groups have noisy estimates, while broad averages can mask systematically poor service. Show sample sizes and uncertainty, protect sensitive data, and define a minimum evidence threshold before interpreting group differences. If a group has too few examples, collect more data or use a cautious evaluation method rather than hiding the result.

Choose a primary metric before final model comparison and tie it to the action the model supports. Keep secondary metrics for tails, scale, and subgroup behavior. Include a baseline, a residual plot, and the metric's known edge cases. This makes the score a useful decision summary rather than a single number detached from operational cost.