Every decision based on uncertain data can go wrong in two ways: believing in an effect that is not there, or failing to see an effect that is. A fire alarm that goes off without smoke and one that stays silent during a fire are both faulty, but in different ways and with different costs. In statistical tests these two errors have precise names, and keeping both under control is the key to designing experiments that are actually useful.
The four possible outcomes
| H₀ true (no effect) | H₀ false (real effect) | |
|---|---|---|
| Do not reject H₀ | Correct decision | Type II error (false negative) |
| Reject H₀ | Type I error (false positive) | Correct decision |
- The probability of a type I error is written α and is the chosen significance level, often 0.05.
- The probability of a type II error is written β.
- The power of the test is 1 − β: the probability of detecting an effect when it really exists.
The trade-off between the two errors
With the same data, reducing one error increases the other. If you make the threshold stricter, moving from α = 0.05 to α = 0.01, you will have fewer false positives but miss more real effects. It is the same dilemma as a spam filter: making it more aggressive blocks more spam but also more legitimate email.
The only way to reduce both errors at the same time is to have more information: a larger sample, less noisy measurements, a better experimental design.
What power depends on
The power of a test increases when:
- the real effect is large: wide differences are easy to distinguish from noise;
- the sample is large: the standard error shrinks with √n;
- the variability of the data is low: precise measurements make even small effects visible;
- α is higher: accepting more false positives means missing fewer true effects.
Of these factors, the one you can usually act on is the second. This is why the sample size must be decided before starting.
Computing the sample size
Consider an A/B test on a page that currently converts 5% of visitors. We want to be able to detect an improvement up to 6%, with two-sided α = 0.05 and 80% power. An approximate formula for comparing two proportions is:
n per group ≈ (z₁₋α/₂ + z₁₋β)² × (p₁(1 − p₁) + p₂(1 − p₂)) / (p₁ − p₂)²
With z₁₋α/₂ ≈ 1.96 and z₁₋β ≈ 0.84:
- (1.96 + 0.84)² ≈ 7.84;
- 0.05 × 0.95 + 0.06 × 0.94 = 0.0475 + 0.0564 = 0.1039;
- (0.01)² = 0.0001;
- n ≈ 7.84 × 0.1039 / 0.0001 ≈ 8,150 visitors per group.
If we wanted to detect a smaller improvement, from 5% to 5.5%, we would need about 31,000 visitors per group: halving the effect to be detected requires about four times the data.
from math import ceil
def n_per_group(p1, p2, z_a=1.96, z_b=0.84):
var = p1 * (1 - p1) + p2 * (1 - p2)
return ceil((z_a + z_b) ** 2 * var / (p1 - p2) ** 2)
print(n_per_group(0.05, 0.06))
print(n_per_group(0.05, 0.055))
The consequences of underpowered tests
An underpowered test is not just a test that «finds nothing». It has subtler and more dangerous consequences.
Negative results that say nothing
If a test has 20% power, a non-significant result is almost devoid of information: even if the effect existed, four times out of five we would not see it. Concluding «the change does not work» is unjustified.
Exaggerated effects when they are found
When an underpowered test turns out significant, the estimate of the effect tends to be inflated. The reason is that, with a lot of noise, the samples that cross the threshold are mainly those in which chance has amplified the effect. It is one of the reasons why many spectacular results are not confirmed by later, larger studies.
More false positives among significant results
If many underpowered tests are run, the share of significant results that are actually false positives increases. The reasoning is the same as Bayes' theorem applied to diagnostic tests: when true effects are rare and sensitivity is low, false alarms weigh heavily.
Do not peek at results along the way
A common mistake in A/B tests is to check the results every day and stop as soon as significance appears. Every check is one more opportunity for a false positive, and the real probability of a type I error becomes much higher than α. If you need to monitor along the way, you must use sequential methods designed for the purpose, which correct the thresholds.
A checklist before the test
- Define the primary metric and the minimum effect you really care about.
- Choose α and the desired power, usually 80% or 90%.
- Compute the sample size and estimate how long it will take to collect it.
- If the time is unrealistic, revise the goal instead of running a test that cannot answer.
- Fix the duration in advance and stick to it.
In short
- A type I error is a false positive, a type II error a false negative.
- Power is the probability of detecting an effect that really exists.
- Reducing both errors requires more data or better data.
- Halving the minimum detectable effect requires about four times the sample.
- Underpowered tests produce empty negative results and exaggerated positive effects.
