In the months when more ice cream is sold, more drownings are also recorded. Nobody thinks ice cream causes drowning: both increase in summer, when it is hot and more people go swimming. The example is trivial, but the same mechanism reappears in far less obvious forms in medicine, economics, marketing and the analysis of computer systems. Knowing the alternative explanations of a correlation is the first step towards not drawing the wrong conclusions.
Five explanations for a correlation
When two quantities A and B turn out to be correlated, the main possibilities are these.
1. A causes B
This is the explanation one usually wants to prove: a new cache configuration reduces response times.
2. B causes A: reverse causation
The direction is the opposite of what is assumed. Teams that use more monitoring tools seem to have more incidents: perhaps it is not monitoring that creates them, but teams with more incidents that have adopted more monitoring. Or monitoring simply uncovers incidents that go unnoticed elsewhere.
3. A common cause: the confounder
A third variable C influences both A and B. This is the case of ice cream and drownings, with temperature as the confounder. In a technical context: older servers have more failures and also older versions of the operating system. Concluding that the operating system causes the failures ignores the age of the hardware, which acts on both.
4. Chance
Searching among very many pairs of variables, some will turn out correlated by pure coincidence. With thousands of system metrics being monitored, you will always find two series that move together without any link. The more you search, the more spurious correlations you find.
5. Data selection
The way the data were collected can create a correlation that does not exist in the population. A classic example is Berkson's bias: if only hospitalised patients are studied, two diseases that are independent in the population can appear negatively correlated, because having one is enough to be admitted. In general, filtering data on a common consequence of A and B introduces artificial associations.
The role of randomised experiments
The most solid way to establish a causal link is the randomised experiment: the treatment is assigned at random. Randomisation makes the groups, on average, similar in everything, including the characteristics we have not measured or thought of. If the groups then differ in outcome, the only systematic difference is the treatment.
A/B tests are randomised experiments: each visitor sees version A or version B at random. This is why, if well run, they allow a difference in conversion to be attributed to the change and not to external factors such as the season or the type of traffic.
When an experiment is not possible
Often you cannot randomise: for ethical or practical reasons, or because the data have already been collected. There are then methods for getting closer to a causal conclusion, with stronger assumptions:
- controlling for known confounders, comparing units that are similar in age, size or type, or including these variables in a statistical model;
- natural experiments, in which an external event assigns the treatment in an almost random way, for example a change applied to only one region for technical reasons;
- difference in differences, comparing the trend before and after an intervention in a treated group and in a control group;
- causal diagrams, graphs that explicitly represent assumptions about what influences what and help decide which variables to control for and which not.
This last point is important: controlling for everything is not a good strategy. Controlling for a variable that lies on the causal path between A and B, or for a common consequence of both, can introduce bias instead of removing it.
Common-sense criteria
In 1965 the epidemiologist Austin Bradford Hill proposed a list of considerations for assessing whether an observed association might be causal. They are not a mechanical checklist, but questions that are useful outside medicine too:
- is the association strong?
- is it consistent across different studies, contexts and data?
- does the cause precede the effect in time?
- is there a dose-response relationship, meaning more cause produces more effect?
- is there a plausible mechanism?
- does an experiment, where possible, confirm it?
A practical checklist
Faced with an interesting correlation:
- ask which causal direction is plausible and whether it might be reversed;
- list the possible common causes and check whether they were measured;
- check how the data were selected;
- ask how many other correlations were searched for before finding this one;
- if possible, design an experiment that separates the explanations.
In short
- A correlation can arise from direct causation, reverse causation, confounders, chance or data selection.
- Confounders are common causes that make two quantities vary together.
- Selecting data on a common consequence creates artificial associations.
- Randomisation is the most solid method for establishing a causal link.
- Without an experiment you need explicit assumptions, targeted controls and plausibility criteria.
