Data Transformation
Transform data (Box-Cox, Johnson, Log, …) for normality
Overview
Data transformation reshapes measurements with a mathematical function (e.g. ln, √, Box-Cox) so that the transformed values come closer to a normal distribution. The point is not to manufacture "better" data, but to make methods applicable that assume normality — Cp/Cpk, I-MR charts, t-tests or linear regression.
A transformation is a tool, not a cure-all. It helps with one class of distribution problems and actively harms another. The first question is always: why are the data not normal?
Available transformations
Box-Cox: (x^λ − 1) / λ with automatically optimized λ. Works only for strictly positive data. Very general-purpose for right-skewed measurands — the most common choice in practice.
Yeo-Johnson: Extension of Box-Cox that also accepts zero and negative values. The right choice when the data can change sign.
Logarithm (ln, log₁₀): Special case of Box-Cox with λ = 0. Classic for lifetimes, concentrations and quantities that behave multiplicatively. Requires positive values.
Square root: Milder correction for slightly right-skewed data or count data (Poisson-like). Values ≥ 0 required.
Inverse (1/x): Very strong compression, suitable for extremely right-skewed data. Values must be non-zero and the sign flips — interpretation becomes awkward.
Square (x²): Stretches the right and compresses the left side — useful for left-skewed distributions, but rarely needed in practice.
Johnson system (SU, SB, SL): Family of very flexible transformations that can normalize almost any continuous distribution. Powerful but complex and hard to communicate — sensible when Box-Cox/Yeo-Johnson are not enough.
When is a transformation useful?
Transform when all three conditions hold: (1) the downstream method assumes normality, (2) your data demonstrably are not normal, and (3) the deviation comes from the shape of the distribution — not from outliers, mixtures or stratification.
Typical use cases
- Process capability (Cp, Cpk, Pp, Ppk) for right-skewed measurands like cycle times, waiting times, surface roughness or lifetimes.
- I-MR or X̄-S charts, when skewness systematically pushes the control limits and produces frequent false alarms.
- Linear regression, when residual analysis shows funnel shape (heteroscedasticity) or skewness — transforming the response often stabilizes variance.
- t-test or ANOVA on heavily skewed data with small samples, where the confidence interval would otherwise be asymmetrically biased.
- Data with a hard physical lower bound at zero (concentrations, particle sizes, defect counts per unit) that become naturally symmetric on a log or square-root scale.
Rule of thumb: if Anderson-Darling or Shapiro-Wilk rejects normality AND the histogram / probability plot shows a clear one-sided skewness, a transformation is worth trying.
When should you NOT transform?
A transformation treats the symptom, not the cause. In the following situations it hides the real problem or solves nothing at all:
Data are already normal: Then there is nothing to do. An unnecessary transformation makes results harder to interpret without improving anything.
Mixed populations / stratification: Several machines, shifts, batches or materials in one dataset cause bimodality or kinks in the probability plot. No λ helps here — split the groups and analyse them separately.
Outliers and special causes: A few extreme values can skew the distribution, but they signal a disturbed process. Investigate and fix the root cause; do not "smooth them away" with a transformation.
Discrete or count data: Counts, categories, ratings or pass/fail data belong to attribute methods (p-, np-, c-, u-charts, Poisson/binomial distribution). Applying Box-Cox to them is methodologically wrong.
Very small samples (n < 20–30): With few observations the normality test itself is unreliable. A transformation then easily fits the random noise and does not generalize to new data.
A non-parametric alternative exists: Wilcoxon, Mann-Whitney, Kruskal-Wallis, Levene or a capability analysis using the actually fitting distribution (Weibull, Lognormal) usually deliver more honest results than a forced normalization.
Communication with stakeholders: Specification limits, tolerances and process targets are defined in original units. If the result has to be understood by customers, auditors or production, intelligibility usually outweighs gaining a few decimals in Cpk.
Short rule: never transform without first looking at the histogram. If you see two peaks, a long tail of three points, or a stair-step pattern from discrete values, transformation is the wrong answer.
Step-by-step workflow
- Inspect histogram and probability plot of the original data — identify shape, skewness and possible mixtures.
- Rule out stratification, outliers and data errors before transforming anything.
- Pick a suitable transformation: positive data with skew → Box-Cox; sign changes → Yeo-Johnson; theoretically motivated (e.g. lifetimes) → logarithm.
- For Box-Cox / Yeo-Johnson use the automatic λ optimization.
- Re-check the transformed data: histogram, probability plot, Anderson-Darling / Shapiro-Wilk.
- Run the downstream analysis (Cpk, control chart, test) on the transformed values.
- Back-transform results into the original unit for communication — carry the specification limits along.
Pitfalls
- Negative or zero values with Box-Cox / logarithm: no result. Use Yeo-Johnson instead, or document an explicit shift.
- Setting λ manually: almost always produces worse results than the automatic optimization — and often hides the fact that the data are not transformable at all.
- Forgetting to transform the specification limits: a Cpk on transformed data is meaningless if USL/LSL still live in the original unit.
- Over-claiming "p > 0.05": passing a normality test on transformed data does not mean the modelling assumptions of the downstream method are truly met. Continue to inspect residuals.
- Selling a transformation as a "data correction": a transformation does not change the values, it changes the scale. This must be communicated clearly for audits, reporting and traceability.
- Reusing the same λ for every new batch: when the process changes, the optimal transformation changes too. Re-calibrate regularly.
Examples
This module ships with the following example datasets — load any of them in the app with a single click.