Regression & Correlation

Regression Coefficients (t-Test & Confidence Intervals)

Tests the significance of individual regression coefficients via t-test and determines confidence intervals.

Statusvalidated

Version1.0.0

Minitab equivalentStat > Regression > Regression > Fit Regression Model (Coefficients table)

Description

For each regression coefficient βⱼ, a t-test checks whether the coefficient is significantly different from zero (H₀: βⱼ = 0). The standard error SE(βⱼ) is derived from the diagonal of the variance-covariance matrix of the estimators. The t-statistic tⱼ = βⱼ / SE(βⱼ) follows a t-distribution with n − p degrees of freedom. A p-value < α indicates that the predictor makes a significant contribution to the model. The confidence intervals [βⱼ ± t_{α/2} · SE(βⱼ)] specify the range in which the true coefficient lies at the chosen confidence level.

Formulas

Schätzer

\hat{β} = (X^{⊤} X)^{- 1} X^{⊤} y

OLS estimation (Ordinary Least Squares): coefficient vector by minimizing the residual sum of squares

Varianz-Kovarianz

Var (\hat{β}) = M S_{E} \cdot (X^{⊤} X)^{- 1}

Variance-covariance matrix of estimators: MSE × inverse of information matrix

SE (\hat{β}_{j}) = M S_{E} \cdot [(X^{⊤} X)^{- 1}]_{jj}

Standard error of the j-th coefficient: square root of MSE × j-th diagonal element of the inverse

t-Statistik

t_{j} = \frac{β ^ _{j}}{SE ( β ^ _{j} )} \sim t (n - p)

t-value: coefficient divided by its standard error. Follows a t-distribution with n − p degrees of freedom under H₀: βⱼ = 0.

p-Wert

p_{j} = 2 \cdot P (T > ∣ t_{j} ∣), T \sim t (n - p)

Two-sided p-value: probability of observing a t-value at least as extreme under H₀

\hat{β}_{j} \pm t_{α /2, n - p} \cdot SE (\hat{β}_{j})

Confidence interval: the true coefficient lies within this range with (1 − α) · 100% probability. If the interval contains zero, the coefficient is not significant.

Assumptions

Residuals are normally distributed (for exact t-distribution)
Homoscedasticity: constant residual variance across all observations
No perfect multicollinearity (X'X must be invertible)
Independence of observations

Limitations

With strong multicollinearity (high VIF), SEs are inflated and t-tests lose power
With non-normal residuals, p-values are only asymptotically correct (large n)
Individual t-tests do not account for multiple testing — with many terms, the probability of false positives increases

References

Montgomery, D.C., Peck, E.A., Vining, G.G., Introduction to Linear Regression Analysis, 6th Ed., Wiley — Chapter 2.4 & 3.3
Kutner, M.H. et al., Applied Linear Statistical Models, 5th Ed., McGraw-Hill — Chapter 6.5
NIST/SEMATECH e-Handbook of Statistical Methods, Section 4.3.1