Polynomial Regression and Underfitting

Add curvature while retaining a linear model

Polynomial regression expands a numeric feature into powers such as x, x squared, and x cubed, then fits a linear model over those transformed columns. The resulting curve can represent some nonlinear relationships while remaining linear in its coefficients. For example, a quadratic model predicts y = beta0 + beta1 x + beta2 x squared. The feature transform adds flexibility; it does not guarantee that a polynomial is the correct shape for the data.

Underfitting occurs when the model cannot capture important structure, leaving systematic residual patterns and poor performance on both training and validation data. A straight line may underfit a clear smooth curve. Increasing the polynomial degree can reduce training error, but high-degree polynomials can oscillate, especially near the ends of the observed range. This raises variance and makes predictions sensitive to small data changes.

Select complexity using held-out evidence

Treat degree as a hyperparameter. Split data according to its deployment structure, then use cross-validation on the training portion to compare candidate degrees. Keep a final test set untouched until model selection is complete. A lower training error is expected as features are added and does not show that a higher degree generalizes better. Compare validation error and inspect residuals across the predictor range.

Polynomial features can be highly correlated. x, x squared, and x cubed may have very different scales, and raw powers can create numerical conditioning problems. Centering or scaling the original feature before expansion can improve stability; use a pipeline so each transformation is fit only on training folds. Scaling the expanded terms can also help regularized models, but document the transformation because coefficients then no longer use the original feature units.

Control extrapolation

A polynomial that fits within the observed range can behave wildly outside it. High-degree curves may turn sharply or diverge, producing impossible predictions. Plot predictions over the training range and beyond it, and define a supported input range. If production inputs can exceed that range, collect data there or choose a model with behavior appropriate to the domain. A curve should not be trusted merely because it passes close to training points.

Polynomial regression can model smooth curvature but does not automatically handle discontinuities, interactions among many features, or changing regimes. Splines or tree-based models may provide more local flexibility. Domain transformations such as a logarithm can be more interpretable when the relationship follows diminishing returns. Choose a model based on the observed residual structure and deployment needs, not on a desire to fit every training point.

Bias, variance, and underfitting

Training and validation errors together help diagnose the balance. High errors on both often indicate underfitting, poor features, or an irreducible target. Low training error with much higher validation error suggests overfitting or a split that does not represent deployment. A degree increase is one response to underfitting, but first verify data quality, target definition, and feature availability. More terms cannot fix labels that are noisy or a prediction task that changes over time.

The useful workflow is to establish a simple baseline, inspect residuals, increase flexibility in controlled steps, and use cross-validation to choose complexity. Preserve uncertainty and supported range in the model documentation. This makes polynomial regression a measured extension of linear regression rather than a way to force a visually smooth curve through every observation.

Model a curved relationship

Assume a sensor's response rises quickly at low concentration and then levels off. Plot response against concentration before fitting. If the pattern is smoothly curved over the observed range, compare a linear baseline with polynomial degrees two through four. Split by time or device if future predictions will involve later periods or unseen devices. Random splitting of repeated readings from the same device can make validation too optimistic.

Use a preprocessing pipeline that centers and scales the input, expands polynomial features, and fits a regressor. Fit the entire pipeline inside cross-validation so the scaler and feature basis never see validation values. Record the degree, random or temporal split, and metric. Compare MAE or RMSE against the baseline and inspect residuals by concentration range. A small metric improvement that varies wildly across folds is not strong evidence for a more complex model.

Inspect stability and shape

Plot the fitted curve with a confidence or prediction band if the inference method supports it. Look for oscillation and sharp edge behavior. Repeat the fit on bootstrap or cross-validation samples to see how much the curve changes. If a high-degree model swings between folds, lower the degree or regularize. If a quadratic still leaves a systematic threshold pattern, a piecewise or domain-specific model may be more appropriate.

Standard polynomial terms can be correlated. Center the feature before expanding so powers have a more stable numeric range. Regularization can shrink coefficients, but it requires scaling and a validation strategy for the penalty strength. Do not interpret raw high-degree coefficients individually as simple physical effects; their meaning depends on the feature origin and other terms. Plot predicted behavior over realistic inputs instead.

Check deployment range

Mark the minimum and maximum training concentrations. Evaluate predictions just inside and outside those limits. Decide how the service handles an input beyond the observed range: warn, reject, fall back to a bounded model, or request additional measurement. Do not silently return a large polynomial extrapolation as if it had equal support. Monitor the proportion of production inputs outside the training range.

Compare the polynomial to alternatives such as a log transform or spline using the same split and metric. Keep the simplest model that satisfies the error tolerance and shape constraints. If response is physically bounded, encode or monitor that constraint; an unconstrained polynomial may predict a negative concentration response even if its average error is low.

Diagnose underfitting carefully

If both train and validation errors are high, confirm the label quality and measurement process before raising degree. The feature may be insufficient to predict the response, or the target may have multiple regimes. Add a scientifically justified feature or collect data that distinguishes the regimes. If train error is low and validation error is high, more degree will usually worsen the generalization gap; use regularization, reduce complexity, or revisit the split.

Write down why the chosen degree is adequate, which alternatives were tested, the observed validation range, and known failure regions. A model card or experiment record should also name the supported feature range and retraining trigger. That evidence helps future users distinguish a validated interpolation from an unsupported extrapolation.

Flexibility has a stability cost

Increasing polynomial degree can reduce bias but increases variance and edge sensitivity. A high-degree model may achieve a tiny training error while producing unrealistic values for nearby inputs. Cross-validation estimates generalization only when its split reflects the deployment population. If observations are temporal, grouped, or repeated per entity, choose a matching split rather than random rows.

Feature expansion can increase dimensionality quickly when several original features are combined. Pairwise interactions and powers can create many terms, increasing compute, collinearity, and explanation burden. Use a degree appropriate to the problem and consider splines, domain transformations, or tree models when they offer a more stable shape. Regularization can help, but it does not remove the need to validate the basis and supported range.

Avoid reading coefficients out of context

The coefficient on x squared is not a standalone effect of curvature independent of the feature's scale and origin. Centering changes coefficient interpretation but can improve numerical behavior. Report predictions or marginal effects over meaningful values rather than quoting a high-order coefficient without context. If the model is used for a scientific claim, consider whether a parametric form reflects a plausible mechanism and whether uncertainty is represented.

Underfitting is not solved by complexity alone. Weak features, measurement error, changing data, and an incorrect target can all cause high error. Compare training and validation performance, inspect residual structure, and confirm that the validation set reflects the intended use. If training error remains high, test a justified feature or model form; if only validation error is high, reduce variance or address distribution shift.

Interview framing

In an interview, explain polynomial features as a basis expansion that remains linear in coefficients. Describe how you select degree with cross-validation, avoid leakage with a pipeline, and inspect residuals and edge behavior. Explain the bias-variance trade-off and why extrapolation is risky. Mention a spline or transformation as an alternative when a global polynomial is unstable. A strong answer ties complexity to evidence and a supported input range instead of choosing a degree because it fits the training plot.

A useful comparison includes a simpler model and at least one alternative that encodes the domain differently. A quadratic term is easy to fit, but a spline can limit local behavior and a logarithmic transform may reflect diminishing returns more naturally. Compare models using the same folds and preprocessing, then examine shape constraints and error in consequential regions. If several models have similar validation performance, prefer the one that is easier to explain, monitor, and maintain.

When a model is retrained, preserve the previous artifact and its evaluation so a new curve can be compared against production behavior. Do not select a degree by looking repeatedly at the final test plot. That turns the test set into a tuning instrument. Define the selection procedure before examining final test outcomes, and record all candidate degrees and folds. Stability across samples is evidence that the selected shape is not an accident of one split.

For an interview, connect underfitting to both train and validation behavior, then explain why adding degree can reduce bias but increase variance. Describe the pipeline that prevents leakage, a residual diagnostic, and an extrapolation limit. Mention an alternative basis when a global polynomial is inappropriate. This makes the model choice a reasoned response to observed structure rather than a visual fit.