A coin tossed 100 times gives 60 heads. Is it loaded, or is it just chance? The new version of a page converts 5.4% of visitors against 5.0% for the old one: is it really better? Hypothesis tests are the classical method for answering questions like these. They are extremely useful, but their most famous output, the p-value, is also one of the most misunderstood concepts in all of statistics.
The logic of the test
A hypothesis test works like a proof by contradiction, in probabilistic form.
- You state a null hypothesis H₀, usually «there is no effect»: the coin is fair, the two pages convert in the same way.
- You state an alternative hypothesis H₁: the coin is not fair, the pages are different.
- You choose a test statistic that measures how far the data stray from what H₀ predicts.
- You compute how likely it would be to observe data this extreme, or more extreme, if H₀ were true.
- If this probability is very small, you conclude that the data are poorly compatible with H₀ and reject it.
The p-value
The p-value is exactly the probability computed in step 4: the probability of obtaining a result at least as extreme as the one observed, assuming the null hypothesis is true.
The coin example
With a fair coin, the number of heads in 100 tosses follows a binomial with n = 100 and p = 0.5. The probability of getting 60 or more heads is about 0.028. Since 40 or fewer heads would be equally suspicious, in a two-sided test you double it: the p-value is about 0.057.
from scipy.stats import binomtest
print(binomtest(60, n=100, p=0.5).pvalue) # about 0.057
With the conventional threshold α = 0.05, this result does not lead to rejecting the hypothesis that the coin is fair. It does not mean the coin is certainly honest: it means that 60 heads out of 100 is not strong enough evidence against this hypothesis. With 1,000 tosses and 600 heads, the same proportion would instead give a tiny p-value.
The significance level
The threshold α is set before looking at the data. It is the probability of rejecting H₀ when it is actually true, that is, the false-alarm rate you are willing to accept. The value 0.05 is a historical convention, not a law of nature: particle physics uses much stricter thresholds, exploratory analyses sometimes looser ones. What matters is to choose it beforehand and not adjust it to the result.
The wrong interpretations
In 2016 the American Statistical Association published an official statement precisely to clear up these misunderstandings. The most common are:
- «The p-value is the probability that H₀ is true.» No. The p-value is computed assuming H₀ is true; it cannot say how probable it is. That would require a Bayesian setup with a prior probability.
- «A p-value of 0.03 means there is a 97% probability that the effect is real.» No, for the same reason.
- «p > 0.05 proves there is no effect.» No. Absence of evidence is not evidence of absence: with little data even important effects may fail to be significant.
- «A small p-value indicates a large effect.» No. With huge samples even practically insignificant differences produce tiny p-values.
- «p = 0.049 and p = 0.051 are opposite results.» No. They are practically the same evidence; the threshold is an arbitrary line drawn on a continuous scale.
Statistical significance and practical importance
Suppose we compare two pages with a million visitors each and find conversion rates of 5.00% and 5.10%. The difference may turn out statistically significant, but is it worth the development cost? The test only answers the question «is the difference distinguishable from chance?», not «is the difference large enough to matter?».
This is why a test should always be accompanied by:
- the effect size, that is, how large the difference is, in understandable units;
- a confidence interval for the effect, showing its uncertainty;
- an assessment of the context: costs, risks, plausibility of the mechanism.
One-sided and two-sided tests
A two-sided test considers extremes in both directions; a one-sided test in only one. The one-sided test has a p-value of about half, and is therefore tempting. It should be used only if, before seeing the data, one of the two directions was truly irrelevant. Choosing it afterwards, because the two-sided result was not significant, is a form of manipulation.
How to write up a result
Instead of «the result is significant (p < 0.05)», it is more informative to write: «the new page converts 0.4 percentage points more (95% CI: from 0.1 to 0.7 points; p = 0.004; 50,000 visitors per group)». The reader understands how large the effect is, how uncertain it is and what data it rests on.
In short
- A hypothesis test assesses how compatible the data are with a null hypothesis.
- The p-value is the probability of data at least this extreme assuming the null hypothesis is true.
- The p-value is not the probability that the null hypothesis is true.
- Statistical significance and practical importance are different things.
- Always report effect size, confidence interval and sample size.
