Regression Analysis
Polynomial regression (degree 1–3, interactions), Exponential, Logarithmic, Power
Overview
Regression describes the relation between a dependent variable (Y, response) and one or more independent variables (X, predictors) through a mathematical model. Where correlation only asks "do they move together?", regression delivers an equation that lets Y be predicted or understood.
Simple linear regression: One X, one Y. Model: Y = β₀ + β₁·X + ε. β₀ is the intercept, β₁ the slope, ε the residual error. The most common entry point.
Multiple regression: Several X variables: Y = β₀ + β₁·X₁ + β₂·X₂ + … + ε. Allows estimating the effect of each variable while holding the others constant.
R² (coefficient of determination): Share of Y variation explained by the model. 0 = model explains nothing, 1 = perfect explanation. R² = 0.85 means: 85% of Y variation is explained by the X.
Adjusted R²: Corrects R² for the number of variables. Important in multiple regression: adding X always raises R², but adjusted R² only rises if they truly contribute.
p-value per coefficient: Tests whether a coefficient differs significantly from 0. p < 0.05 → the variable demonstrably contributes to the model.
Residuals: Difference between observed Y and predicted Y. A good regression has randomly scattered residuals without pattern — patterns indicate missed effects.
Approach
- Before regression, make scatter plots — do the data fit a linear model?
- Fit the model (pick Y and X, run the software).
- Check R² and p-values — does the model make sense at all?
- Inspect residuals: random scatter around zero, no patterns, no funnels.
- Apply the model to new data — does it actually predict what it should?
- For multiple regression, check multicollinearity (VIF) and drop redundant X.
Interpretation
A high R² is not automatically a good model. More important: do predictions hold for values outside the training data, are residuals well-behaved, and does the model make domain sense? A model with R² = 0.99 and systematic residuals is worth less than one with R² = 0.7 and clean residuals.
Predictions outside the observed X range (extrapolation) are risky. The model "knows" nothing about what it has not seen — the linear relation may have ended long ago.
Pitfalls
Linear model for nonlinear data: A curved relation is only roughly captured by a straight line. Residuals show clear curvature — that is the signal to try a nonlinear or transformed model.
Multicollinearity: When two X are strongly correlated, the model cannot separate their individual effects — coefficients become unstable and hard to interpret. VIF > 5 is a warning sign.
Overfitting: A model with too many X for too few observations fits random noise. R² looks great, new data disappoints. Rule of thumb: at least 10 observations per coefficient.
Heteroscedasticity ignored: When residual scatter grows with X (funnel shape), standard errors are biased — p-values and confidence intervals lie. Transformation or weighted regression helps.
Extrapolation: Using the model outside the training range — risky because nothing has been validated there. Treat predictions as reliable only inside the observed X range.
Causal language on observational data: A regression on observational data describes relations but does not prove causation. Causal claims need an experiment (DOE) or very strong arguments.
Examples
This module ships with the following example datasets — load any of them in the app with a single click.
Available in the following cycles
- DMAIC: Improve
- DMADV: Design
- 8D: D5 — Corrective Actions