Regression & Correlation

Regression ANOVA (Variance Decomposition)

Decomposes total variation of the response into explained (regression) and unexplained (residual) components.

Statusvalidated

Version1.0.0

Minitab equivalentStat > Regression > Regression > Fit Regression Model (ANOVA table)

Description

Regression ANOVA tests whether the regression model as a whole contributes significantly to explaining the variance of the response variable. The total variation SST is decomposed into the model-explained variation SSR and the residual variation SSE. Through degrees of freedom (df), mean squares (MS) are calculated, whose ratio yields the F-statistic. A significant F-test (p < α) indicates that at least one predictor has a statistically meaningful effect on the response.

Formulas

SST

S S_{T} = i = 1 \sum n (y_{i} - \overset{y}{ˉ})^{2}

Total Sum of Squares: sum of squared deviations of all observations from the overall mean

SSR

S S_{R} = i = 1 \sum n (\overset{y}{^}_{i} - \overset{y}{ˉ})^{2}

Regression Sum of Squares: variation explained by the model

SSE

S S_{E} = i = 1 \sum n (y_{i} - \overset{y}{^}_{i})^{2} = i = 1 \sum n e_{i}^{2}

Error Sum of Squares: unexplained variation (residuals)

Zerlegung

S S_{T} = S S_{R} + S S_{E}

Fundamental identity: total variation = model variation + residual variation

d f_{R} = p - 1, d f_{E} = n - p, d f_{T} = n - 1

Degrees of freedom: p = number of model parameters (incl. intercept), n = sample size

M S_{R} = \frac{S S _{R}}{d f _{R}}, M S_{E} = \frac{S S _{E}}{d f _{E}}

Mean Squares: SS normalized by degrees of freedom

F = \frac{M S _{R}}{M S _{E}} \sim F (d f_{R}, d f_{E})

F-statistic: ratio of mean model variation to mean residual variation. Follows an F-distribution under H₀.

R²

R^{2} = 1 - \frac{S S _{E}}{S S _{T}} = \frac{S S _{R}}{S S _{T}}

Coefficient of determination: proportion of variance explained by the model (0 ≤ R² ≤ 1)

Assumptions

Linear relationship between predictors and response (in parameter space)
Residuals are normally distributed with mean 0
Homoscedasticity: constant residual variance
Independence of observations

Limitations

A significant F-test does not indicate which predictors are significant — check the coefficient t-tests for that
R² increases with every additional variable even without real contribution — prefer Adj. R²

References

Montgomery, D.C., Peck, E.A., Vining, G.G., Introduction to Linear Regression Analysis, 6th Ed., Wiley — Chapter 2 & 3
Kutner, M.H. et al., Applied Linear Statistical Models, 5th Ed., McGraw-Hill — Chapter 6
NIST/SEMATECH e-Handbook of Statistical Methods, Section 4.2.1