Regression & Correlation

Logistic Regression (Binomial GLM)

Models a binary response (0/1) as a function of predictors using the logistic link function and IRLS estimation.

Statusvalidated

Version1.0.0

Minitab equivalentStat > Regression > Binary Logistic Regression > Fit Binary Logistic Model

Description

Logistic regression is a generalized linear model (GLM) with binomial distribution and logit link function. The probability P(Y = 1 | X) is modeled via the inverse logit function: μ = 1 / (1 + exp(−X'β)). Coefficients are estimated iteratively via IRLS (Iteratively Reweighted Least Squares), solving weighted normal equations in each step with working responses z and weights w = μ(1−μ). The significance of individual coefficients is tested via Wald z-tests: z_j = β̂_j / SE(β̂_j). Odds ratios exp(β̂_j) quantify the multiplicative effect of a predictor on the odds. Model fit is assessed via deviance, AIC, BIC, and pseudo-R² measures (McFadden, Cox-Snell, Nagelkerke).

Formulas

Link-Funktion

logit (μ) = ln (\frac{μ}{1 - μ}) = X^{⊤} β

Logit link function: transforms the success probability μ onto the real line as a linear combination of predictors

IRLS-Gewichte

w_{i} = \overset{μ}{^}_{i} (1 - \overset{μ}{^}_{i})

IRLS weights: variance of the binomial distribution, used for iteratively reweighted estimation

Arbeitsvariable

z_{i} = \overset{η}{^}_{i} + \frac{y _{i} - μ ^ _{i}}{μ ^ _{i} ( 1 - μ ^ _{i} )}

Working response: linearized pseudo-observation for the IRLS step

Fisher-Information

I (β) = X^{⊤} WX

Fisher information matrix: its inverse yields the variance-covariance matrix of the estimators

Wald z-Statistik

z_{j} = \frac{β ^ _{j}}{SE ( β ^ _{j} )}

Wald z-statistic: coefficient divided by its standard error, asymptotically standard normal under H₀: βⱼ = 0

Odds Ratio

O R_{j} = exp (\hat{β}_{j})

Odds ratio: multiplicative effect of a one-unit change in the j-th predictor on the odds

Devianz

D = 2 i = 1 \sum n [y_{i} ln (\frac{y _{i}}{μ ^ _{i}}) + (1 - y_{i}) ln (\frac{1 - y _{i}}{1 - μ ^ _{i}})]

Deviance: measure of departure from the saturated model (smaller is better)

AIC

AIC = - 2 ℓ + 2 p

Akaike Information Criterion: model fit penalized for complexity (p = number of parameters)

McFadden R²

R_{McF}^{2} = 1 - \frac{ℓ}{ℓ _{0}}

McFadden pseudo-R²: proportion of log-likelihood explained by the model (ℓ₀ = null model)

Assumptions

Binary response variable (0/1)
Independence of observations
Linear relationship between logit(π) and the predictors
No perfect multicollinearity among predictors
Sufficient sample size (rule of thumb: ≥ 10 events per predictor)

Limitations

Complete or quasi-complete separation → infinite β̂ estimates and extremely large standard errors
Not suitable for ordinal or multinomial responses
Sensitive to outliers in predictor space
No overdispersion handling (use quasi-binomial or beta-binomial for that)

References

Hosmer, D.W., Lemeshow, S., Sturdivant, R.X., Applied Logistic Regression, 3rd Ed., Wiley
Agresti, A., Categorical Data Analysis, 3rd Ed., Wiley
McCullagh, P., Nelder, J.A., Generalized Linear Models, 2nd Ed., Chapman & Hall