A report says that the average load time of a page is 820 milliseconds. But that number comes from 50 measurements: if we had taken another 50, would we have obtained the same value? Almost certainly not. A point estimate without an indication of uncertainty is only half the information. The confidence interval is the tool that adds the missing part: a set of plausible values for the parameter we are estimating.
The underlying idea
Every estimate computed on a sample varies from sample to sample. The central limit theorem tells us that, for large enough samples, the sample mean is approximately normally distributed around the true mean, with a standard deviation equal to the standard error. Hence the general form:
estimate ± critical value × standard error
For a 95% confidence level and large samples the critical value is about 1.96.
Example 1: a mean
We measure a page's load time 50 times. We obtain a mean of 820 ms and a sample standard deviation of 150 ms.
- Standard error: 150 / √50 ≈ 21.2 ms.
- Since the standard deviation is estimated from the sample, Student's t distribution with 49 degrees of freedom is used, whose 95% critical value is about 2.01.
- Margin: 2.01 × 21.2 ≈ 42.6 ms.
- Interval: from about 777 to 863 ms.
With small samples the difference between 1.96 and the t value becomes important; with large samples it is negligible.
Example 2: a proportion
Out of 400 visitors to a page, 120 sign up for the newsletter. The estimated proportion is 0.30.
- Standard error: √(0.30 × 0.70 / 400) ≈ 0.0229.
- 95% margin: 1.96 × 0.0229 ≈ 0.045.
- Interval: from about 25.5% to 34.5%.
This simple formula works poorly when the proportion is close to 0 or 1 or the counts are small: with 2 successes out of 50 it can produce a negative lower limit, which is meaningless. In these cases more accurate methods are used, such as the Wilson interval, available in every statistical library.
What «95%» really means
This is the most misunderstood point. The correct interpretation concerns the method, not the single interval: if we repeated the experiment a great many times and computed the interval in the same way each time, about 95% of those intervals would contain the true value.
In the classical approach the true value is fixed and the interval is random. Once a specific interval has been computed, such as 777–863 ms, the true value is either inside it or not. Saying «there is a 95% probability that the true mean lies between 777 and 863» is, strictly speaking, a Bayesian statement that would require a different setup, even though in many practical cases the numbers turn out similar.
Other wrong readings
- «95% of the data fall within the interval»: no. The interval concerns the mean, not the individual values. Individual load times are much more spread out.
- «A larger sample gives an interval that contains the true value more often»: no, the level stays at 95%; the interval does, however, become narrower.
- «If two intervals overlap, the difference is not significant»: not necessarily. Two intervals can partly overlap even when the difference between the means is statistically significant. To compare two groups you must compute the interval of the difference.
What makes the interval narrower
The width depends on three factors:
- the variability of the data: the more spread out they are, the wider the interval;
- the sample size: the width decreases with √n, so halving it requires a sample four times larger;
- the confidence level: a 99% interval is wider than a 95% one, because to «capture» the true value more often it must cover more values.
The bootstrap
When the theoretical formula is not available or the assumptions are doubtful, for example for a median or a percentile, you can use the bootstrap: draw from the sample, with replacement, thousands of new samples of the same size, compute the statistic on each and take the 2.5 and 97.5 percentiles of the results.
import numpy as np
rng = np.random.default_rng(0)
times = np.array([...]) # the measurements collected
estimates = [np.median(rng.choice(times, len(times))) for _ in range(10_000)]
print(np.percentile(estimates, [2.5, 97.5]))
It is a simple and very general method, but it works no miracles: with small or biased samples, the bootstrap faithfully reproduces the same limitations.
How to report them
A good report writes «820 ms (95% CI: 777–863)», states the method used and the sample size. In my work I prefer intervals to significance tests alone, because they show together the direction of the effect, its size and the uncertainty that comes with it.
In short
- A confidence interval adds to the estimate the uncertainty due to sampling.
- The general form is estimate ± critical value × standard error.
- The 95% refers to the method: 95% of intervals built this way contain the true value.
- The interval concerns the parameter, not the individual data, and narrows with √n.
- For complex statistics the bootstrap offers a practical alternative to formulas.
