Simple Linear Regression

Fit a relationship between one feature and a target

Simple linear regression models a continuous target as a straight-line function of one numeric predictor: y = beta0 + beta1 x + error. The intercept beta0 is the predicted target when x is zero, and the slope beta1 is the predicted change in y for a one-unit increase in x. The units matter. A slope of 2.4 dollars per square meter has a different meaning from 2.4 dollars per apartment, and interpreting an intercept outside the observed feature range may have no practical meaning.

Ordinary least squares (OLS) selects coefficients that minimize the sum of squared residuals. A residual is observed y minus predicted y. Squaring penalizes large misses more strongly than small ones and yields a closed-form solution for a simple line, though software usually computes it numerically in a stable way. OLS is a fitting objective, not proof that the relationship is linear or causal.

Read the fit and its assumptions

Plot the data and fitted line before trusting a coefficient. A strong curved pattern suggests the linear form is misspecified. Residuals should not show a systematic curve or funnel shape if the simple assumptions are reasonable. A few influential points can pull the slope substantially; inspect leverage and residual magnitude rather than deleting points automatically. An outlier may be a data error, a rare but valid observation, or evidence that the model needs another feature.

Common inferential assumptions include a correctly specified linear conditional mean, independent errors, and constant error variance for standard OLS uncertainty estimates. Normal residuals are useful for small-sample inference, but they are not required for OLS coefficients to be computable. Violations change interpretation and uncertainty. Time series observations may be correlated; a funnel-shaped residual pattern suggests heteroskedasticity; a curved residual pattern suggests a missing nonlinear term.

The slope describes association conditional on the model, not causality. A line between advertising spend and sales does not prove the spend caused the change; seasonality, pricing, or market size may affect both. A causal claim needs a suitable design or assumptions beyond a fitted regression. Likewise, correlation does not guarantee predictive value on future or held-out examples.

Evaluate on held-out data

Split data according to how predictions will be used. Random splitting may be reasonable for independent observations, but time-ordered data should usually be split chronologically to avoid training on the future. Fit the coefficients only on training data, then calculate metrics on validation or test data. Repeatedly selecting models based on the test set turns it into another training signal.

Mean absolute error reports average absolute miss in the target's units and is less dominated by a few large errors than squared loss. Mean squared error emphasizes large errors; root mean squared error returns to target units. R-squared measures variance explained relative to predicting the training or evaluation-set mean, but a high value does not establish causality, low error for every subgroup, or good extrapolation. Always compare against a simple baseline and inspect residuals by range or relevant cohort.

Know where the line stops helping

The model is most useful as an interpretable baseline and for a roughly linear relationship with a meaningful single predictor. It can be a poor fit when the relationship curves, the target has strong group structure, errors are dependent, or relevant confounders are omitted. Extrapolating far beyond observed x values is especially risky because the fitted line continues indefinitely even when the underlying process cannot.

Transformations can make a relationship more linear, and polynomial terms can represent curvature, but they change interpretation and may overfit. A more flexible model may improve prediction while reducing transparency. Start with the line, inspect its failures, and add complexity only when held-out evidence and domain knowledge support it. A simple model is valuable because its assumptions and errors are easy to see.

Build a training-time baseline

Suppose a team wants to estimate training time from the number of examples in a fixed model configuration. Plot example count against observed runtime and inspect whether the points plausibly follow a line over the measured range. Randomly splitting independent runs can be acceptable, but if runs span hardware or software releases, preserve those groups so near-duplicate conditions do not leak between train and test. Record units and collection conditions before fitting.

Fit an OLS line on the training observations and record the intercept, slope, and residuals. If the slope is 0.003 seconds per example, a 10,000-example increase corresponds to a fitted increase of about 30 seconds within the model's observed range. That interpretation assumes the feature scale and runtime conditions are stable. It does not claim that adding examples always causes a linear runtime increase when batching, cache behavior, or hardware saturation changes.

Compare predictions with a baseline that always predicts the training mean. Calculate MAE and RMSE on held-out data and inspect residuals against predicted values and example count. A funnel pattern may mean variance grows with workload. A curve may indicate fixed setup overhead plus near-linear per-example work, or a hardware threshold. Plotting helps identify whether the line is a useful approximation and where it stops working.

Test influential points and extrapolation

Temporarily refit after checking a suspicious run, but do not remove it solely because it changes the coefficient. Confirm whether the run used the same hardware, whether measurement failed, and whether it represents a real operating condition. Report results with and without a point only when there is a defensible data-quality reason, and preserve the decision in the experiment record.

Try predicting at twice the largest observed example count. Ask whether the model has evidence at that scale and whether the system changes batch size or memory behavior. If a prediction is needed there, collect representative runs or use a model tied to the underlying cost structure. Extrapolation is not validated merely because the equation returns a number.

Use a reproducible implementation

In scikit-learn, use a pipeline when preprocessing is required and fit it only on training data. Keep the split, random seed, feature definition, and library version with the experiment. The fitted model stores an intercept and coefficient; retain the training range and target units alongside them so downstream users do not treat the coefficient as universal. Avoid fitting a scaler on the complete dataset before splitting, which leaks evaluation information.

Check that inputs are numeric, units are consistent, missing observations have an explicit policy, and train and inference code use the same feature construction. Evaluate on a time-aware or group-aware split when the deployment scenario calls for it. Save predictions and residuals for error analysis, but avoid storing sensitive raw examples without a retention and access policy.

Explain the result honestly

Present the fitted line, its evaluation metric, a baseline comparison, and the range where the data supports predictions. Describe the largest residual pattern and one limitation. If the goal is causal explanation, state that regression alone does not establish causation. If the goal is prediction, focus on held-out performance and robustness to workload changes. A useful baseline is one the team can reproduce and challenge, not just a coefficient in a slide.

Interpretability versus fit

Simple linear regression is easy to explain, inexpensive to train, and useful as a baseline. Its transparency can reveal direction, scale, and residual behavior. That does not mean the model is automatically correct or fair. A single predictor may be a proxy for a protected or operationally sensitive attribute, and an apparently interpretable slope can reflect omitted variables. Review feature meaning and use context before presenting a coefficient as an explanation.

The squared-error objective is sensitive to large residuals. If outliers are valid but costly to fit, robust regression or a different loss may be appropriate. If residual variance changes across the feature range, standard errors and prediction intervals need appropriate treatment. If the relationship is curved, a straight line can systematically underpredict one region and overpredict another. Residual analysis is part of model evaluation, not optional decoration.

Use metrics with context

R-squared can increase when predictors are added and can be negative on held-out data. It is not a direct measure of practical usefulness. Compare MAE or RMSE to a baseline and to the cost of prediction errors. A ten-unit error may be acceptable for a rough capacity estimate and unacceptable for a medication dose. Report performance by relevant range and subgroup so an average does not conceal a high-impact failure.

Keep the test set separate from repeated feature selection. If the same held-out set guides every model decision, the reported score becomes optimistic. Use cross-validation on training data for selection, then evaluate once on a final test set or use a deployment-appropriate temporal split. In production, monitor feature range and residuals so distribution changes are visible.

Know when to extend the model

Multiple regression can control for additional measured predictors, but it does not automatically solve confounding or multicollinearity. Polynomial features can model curvature but increase variance and require careful degree selection. Tree-based models can represent nonlinear interactions with less manual feature design, but may be harder to interpret. Choose the next model based on observed failure and evaluation criteria, not a presumption that more complexity is better.

In an interview, explain the OLS objective, slope units, residual diagnostics, and held-out evaluation. Clarify that association is not causation and that extrapolation beyond observed data is risky. Give a case where the simple line is a valuable baseline and one residual pattern that would make you extend it. This demonstrates both understanding of the method and judgment about its limits.

Prediction intervals and confidence intervals answer different questions. A confidence interval around the fitted mean describes uncertainty in the estimated average response at a feature value, while a prediction interval for a new observation also includes individual residual variation and is therefore wider. Neither interval remains trustworthy far outside the training range or under a changed data-generating process. State the population and period to which the evaluation applies.

Monitor the feature range and residual distribution after deployment. If incoming values fall beyond the training range, flag extrapolation rather than presenting an ordinary prediction without context. If the relationship changes over time, retrain only after checking data quality and the intended prediction target. A simple model is easy to refresh, but repeated retraining can hide a process change that deserves investigation.