An A/B test shows no difference on the primary metric. So you look at the other metrics: time on page, menu clicks, newsletter sign-ups. Then the segments: mobile users, desktop users, new visitors, returning visitors, individual countries. In the end, in some combination, a «significant» result appears. Is it a discovery? Almost certainly not. It is the problem of multiple comparisons, one of the main causes of results that fail to replicate.
The problem in one number
With a significance level α = 0.05, each single test has a 5% probability of giving a false positive when there is no effect. But if many independent tests are run, the probability that at least one gives a false positive grows quickly:
| Number of tests | P(at least one false positive) |
|---|---|
| 1 | 5% |
| 5 | about 23% |
| 10 | about 40% |
| 20 | about 64% |
| 50 | about 92% |
The calculation is 1 − 0.95ᵏ, where k is the number of tests. With 10 metrics and 5 segments there are 50 comparisons: even if the change has no effect at all, you expect on average about 2.5 significant results.
P-hacking
P-hacking is the set of practices, often unintentional, that artificially increase the probability of getting a p-value below the threshold. The most common forms are:
- trying many metrics and reporting only the significant one;
- splitting the data into segments until one segment shows an effect;
- stopping the experiment as soon as the result becomes significant;
- excluding outliers only after seeing that they change the result;
- trying different transformations or models and choosing the most favourable;
- changing the hypothesis after seeing the data, presenting a chance finding as predicted.
Each single choice can seem reasonable. The problem is that, when choices depend on the data, the final p-value no longer has its stated meaning. The statisticians Andrew Gelman and Eric Loken called this situation the «garden of forking paths»: even without explicitly running many tests, the many possible decisions amount to multiplying the comparisons.
The corrections
Bonferroni
The simplest correction divides the threshold by the number of tests: with 20 tests, only a p-value below 0.05 / 20 = 0.0025 is considered significant. This keeps the probability of at least one false positive below 5%. It is easy to apply but very conservative: with many tests it sharply reduces power and causes real effects to be missed.
Holm
Holm's procedure is an improved version that controls the same error with more power. The p-values are sorted from smallest to largest; the first is compared with α/k, the second with α/(k − 1), and so on, stopping at the first that does not pass the threshold. It has no drawbacks compared with Bonferroni and should be preferred.
Benjamini-Hochberg
When there are many tests, as in genomics or in the analysis of hundreds of metrics, controlling the probability of even a single false positive is too strict. The Benjamini-Hochberg procedure instead controls the false discovery rate, that is, the expected share of false positives among the results declared significant.
It works like this: sort the k p-values from smallest, compare the p-value in position i with (i / k) · q, where q is the accepted share of false discoveries, for example 0.05 or 0.10, and declare significant all tests up to the last one that meets the condition.
from statsmodels.stats.multitest import multipletests
pvalues = [0.001, 0.008, 0.039, 0.041, 0.042, 0.06, 0.07, 0.2, 0.5, 0.9]
for method in ("bonferroni", "holm", "fdr_bh"):
reject, *_ = multipletests(pvalues, alpha=0.05, method=method)
print(method, reject.sum())
With these values, Bonferroni and Holm keep only one significant result, Benjamini-Hochberg two: the three p-values between 0.039 and 0.042, which individually would look interesting, survive none of the corrections.
Prevention is better than correction
Corrections help, but the most effective solution is to organise the analysis so that you do not need them.
- Define in advance the primary metric, the segments to be analysed and the duration of the test, and write them down. In scientific research this practice is called preregistration.
- Distinguish confirmatory from exploratory analysis. Exploring data is legitimate and useful, but exploratory results should be presented as hypotheses to be tested, not as discoveries.
- Report all the tests performed, not only the significant ones. A reader who knows 50 comparisons were made interprets a single positive result differently.
- Replicate. The most convincing way to confirm an unexpected result is to repeat the experiment with new data, focusing only on that result.
- Look at effect size and plausibility. A huge effect in a small segment that was never hypothesised beforehand deserves a lot of scepticism.
In short
- Every additional test increases the probability of at least one false positive.
- With 20 independent tests and no real effects, that probability is about 64%.
- P-hacking arises from data-driven choices, even without any intention to deceive.
- Holm controls the overall error better than Bonferroni; Benjamini-Hochberg controls the share of false discoveries.
- Defining the analysis in advance and replicating unexpected results is the most solid defence.
