Analyze

Sample Size

Calculate the required sample size for variance and mean tests

Overview

Sample size planning determines how many observations are needed for a test or estimate to have the desired statistical power. Too few samples miss real effects; too many waste resources. Before any serious experiment or comparison, plan the sample size — not after.

Significance level (α): The risk of seeing an effect that does not exist (Type I error). Usually α = 0.05.

Power (1 − β): The probability of detecting an effect that really exists. Common: power = 0.80 or 0.90. Low power means: even a real effect is often missed.

Effect size (δ): The smallest difference still considered practically relevant. "From which difference on would we act?". Smaller effects need larger samples.

Variation (σ): The scatter of individual measurements. Greater variation requires more observations to pull an effect out of the noise.

Confidence interval width: For estimates (not tests), the sample size is chosen so the interval becomes narrow enough to answer the question. Halving the width quadruples n.

The four quantities α, power, δ, and σ are coupled: fix three and the fourth follows. In planning, α, power, and δ are set; σ is estimated from prior knowledge or pilot data — n is the result.

Approach

Clarify test type or estimation problem — 1-sample, 2-sample, proportion comparison, regression?
Choose significance level (usually α = 0.05).
Set power (usually 0.80 or 0.90).
Define the smallest practically relevant effect size — not the expected one, the smallest still interesting.
Estimate variation — from history, pilot data, or literature.
Compute n and check whether it is resource-feasible. If not, honestly revise parameters or redesign the experiment.
Document the planned size — so it is later traceable why exactly this n was used.

Pitfalls

Power too low: Power of 0.50 is a coin flip — even a real effect is found only half the time. Below 0.80 it becomes risky.

Effect size too optimistic: Assuming a large effect gives a comfortable small n. If reality shows only a small effect, the test is blind. Plan pessimistically to avoid empty results.

Variation underestimated: Without pilot data or history, σ is often set too low. Result: n too small, test misses the effect. When uncertain, round up.

Post-hoc power: Computing power after the test with the observed effect is nonsense — it correlates directly with the p-value. Power belongs to planning, not to evaluation.

Sampling design forgotten: Size alone is not enough — how the sample is drawn matters equally. A biased draw cannot be fixed by any formula.

"The more, the better": Unnecessarily large samples inflate cost and produce significant but irrelevant results. Planning deliberately picks just the sample size needed.

Examples

This module ships with the following example datasets — load any of them in the app with a single click.

Detect variance reduction (1-sample) — Project template

After a process improvement, scatter should have dropped from σ₀ = 1.5 to σ₁ = 1.0. One-sided chi-square test, α = 0.05, power = 80 %. Returns the n needed to confirm the improvement statistically.

Compare variances of two suppliers (2-sample) — Project template

Compare scatter of two suppliers: suspected doubled variance (σ²₁/σ²₂ = 2.0). Two-sided F-test, α = 0.05, power = 80 %. Returns the sample size per group.

Mean shift from target (1-sample) — Project template

Check whether the process mean drifts by Δ = 0.3 from target (σ ≈ 0.5). Two-sided t-test, α = 0.05, power = 80 %. Cohen's d = 0.6 (medium effect).

Compare means of two machines (2-sample) — Project template

Compare cycle time of two machines: difference Δ = 0.5 min at σ ≈ 0.8 min. Two-sided t-test, α = 0.05, power = 80 %. Cohen's d ≈ 0.63 — returns the n per group.

Available in the following cycles

DMAIC: Analyze
DMADV: Analyze
8D: D4 — Root Cause Analysis