Specialty › Resampling

Permutation test

A permutation test compares two samples by reshuffling the group labels — distribution-free, and exact when randomisation was part of the design.

What is Permutation test?

Under the strong null (the two groups are exchangeable — the labels are random), any permutation of the group labels gives an equally likely test statistic. The permutation p-value is the proportion of permutations where the recomputed statistic is at least as extreme as the observed.

For two independent samples, permute the group labels. For paired data, randomly flip the signs of the within-pair differences. With enough permutations (R = 5000+), the p-value converges to the exact randomisation p; with all permutations enumerated (computationally feasible only for small n), the test is exact.

The permutation distribution requires the *exchangeability* assumption: under H₀, all permutations of labels are equally likely. For randomised experiments this is by design. For observational data, exchangeability is more questionable and the test is approximate.

When should I use Permutation test?

Randomised experiments — permutation is the natural inference framework.
Small samples where asymptotic tests don't apply.
Test statistics with no analytic distribution (e.g. trimmed-mean difference).

What data does it need?

Two numeric columns + test statistic + R + alternative + paired flag.

What does it report?

Observed statistic, (count + 1)/(R + 1) p-value (continuity-corrected so p > 0 even with R draws all less extreme).

What does it assume?

Exchangeability under H₀.
Independent permutations (R large enough to span the true distribution).

How do I interpret the result?

Permutation p ≈ analytic p when the analytic test's assumptions hold; useful as a sanity check on the t-test in borderline-non-normal data.

References

Good (2005). Permutation, Parametric, and Bootstrap Tests of Hypotheses, 3rd ed.