Analyze

Hypothesis Test

Variance and mean tests with automatic normality assessment and power analysis

Overview

A hypothesis test checks whether an observed deviation in the data is a real effect or could have arisen by chance. It is the central decision instrument of the Analyze phase — used to statistically back up statements like "the mean has changed" or "the two machines deliver different quality".

Null hypothesis (H₀): The "nothing going on" assumption. Typical: "means are equal", "there is no difference". The test tries to refute it — never to confirm it.

Alternative hypothesis (H₁): The interesting counter-assumption: "the means differ", "the new process is better". Accepted when data refute H₀.

p-value: Probability of seeing data this extreme (or more) if H₀ were true. Small = H₀ unlikely. Usual threshold: p < 0.05 → reject H₀.

Significance level (α): The risk threshold, set in advance, for wrongly rejecting H₀. Usually 0.05 (5%). α is an analyst decision, not a data property.

Type I and Type II errors: Type I (α): reject H₀ when it is true — "false alarm". Type II (β): fail to reject H₀ when it is false — "missed real effect". Power = 1 − β.

Test statistic: A quantity computed from the data (e.g. t, z, F, χ²) placed against a known distribution. From it the p-value is derived.

The right test depends on the question (one mean, two groups, more than two?), the scale (continuous, categorical), and distributional assumptions. Common tests: 1-sample t, 2-sample t, paired t, ANOVA, chi-square, Mann-Whitney (nonparametric).

k-sample comparison (means): For more than two groups, one-way analysis of variance (One-Way ANOVA) is the appropriate procedure — it decomposes the total variation into between- and within-group components and compares them via an F-statistic. For non-normal data the module automatically switches to the rank-based Kruskal-Wallis test. Both are omnibus tests: they show that at least one mean differs without identifying which one — that requires follow-up post-hoc comparisons (e.g. Tukey HSD).

k-sample comparison (variances): For equality of variances across several groups, two tests are available: Bartlett (parametric, very powerful under normality but sensitive to departures from it) and Levene (Brown-Forsythe variant, robust to non-normality). The module picks automatically based on the normality assessment.

Approach

State the question precisely — what do I actually want to know?
Write down H₀ and H₁ clearly — before analyzing data.
Set the significance level α (typically 0.05) — before computing.
Choose the right test (based on scale, number of groups, distribution, independence).
Check assumptions (normality? equal variances? independent observations?).
Run the test and read the p-value.
Decide: p < α → reject H₀; otherwise no evidence against H₀.
Additionally report effect size and confidence interval — a p-value alone says little about magnitude and relevance.

Pitfalls

"Accept H₀" instead of "fail to reject": A non-significant result does not prove H₀ — it only shows the data are not enough to refute it. Absence of evidence is not evidence of absence.

Reading the p-value as effect size: p = 0.001 does not mean "large effect", only "very unlikely under H₀". p-values depend heavily on sample size. Effect size and confidence interval show magnitude.

Assumptions not checked: A t-test on strongly skewed data or ANOVA on very unequal variances yields wrong p-values. At least check histogram and variances, switch to nonparametric if needed.

Multiple testing without correction: Running 20 tests on random data produces roughly one with p < 0.05 by chance. For many comparisons use Bonferroni, FDR, or a better design.

Hypothesizing after seeing the data: Writing H₀/H₁ only after looking at the data turns exploration into apparent confirmation (p-hacking). The hypothesis belongs before the experiment.

Significant ≠ practically relevant: At n = 100,000 even the tiniest difference becomes significant. Before presenting always ask: "is the difference big enough to act on?".

Examples

This module ships with the following example datasets — load any of them in the app with a single click.

Delivery time by driver (k-sample ANOVA) — Project template

Delivery times (min) for three drivers A/B/C, 15 each. True means 22/25/28 min at σ ≈ 3 — Cohen's f ≈ 0.82, strong ANOVA signal.

Complaints by branch (5-sample ANOVA, unbalanced) — Project template

Complaints per day across five branches A–E, sample sizes 8/10/12/15/9. Branch D has a clearly elevated mean (6.2 vs. ~5.0) — post-hoc comparisons separate D from the rest.

Available in the following cycles

DMAIC: Analyze
DMADV: Analyze
8D: D4 — Root Cause Analysis