One week a service's response times are terrible. The team steps in, changes a few parameters, and the following week things go better. Did the intervention work? Perhaps. But things might well have improved anyway, because of a statistical phenomenon as common as it is underestimated: regression to the mean.
The phenomenon
Every real measurement is the sum of a stable component and a random component. A student's result in a test depends on their preparation, but also on the day, the questions that came up, tiredness. A service's average response time over a week depends on its architecture, but also on traffic, a temporary failure at a supplier, an unexpected spike.
When we observe an extreme value, it is likely to be extreme partly because the random component pushed in the same direction. At the next measurement the random component is drawn again, and it is unlikely to be just as extreme. The result therefore tends to come back closer to the mean, without anything having changed.
A bit of history
The phenomenon was described by Francis Galton in the nineteenth century while studying the heights of parents and children. Very tall parents tended to have tall children, but on average less tall than themselves; very short parents tended to have short children, but less short than themselves. Galton spoke of «regression towards mediocrity», and that is where the term «regression» still used in statistics today comes from.
How strong the effect is
If two successive measurements have a correlation r, and we express them as z-scores, the expected value of the second measurement, given the value of the first, is:
expected z at the second measurement = r × z at the first measurement
- If r = 1, the measurements are perfectly stable and there is no regression.
- If r = 0, the first measurement says nothing about the second, and you expect to return exactly to the mean.
- With r = 0.5, a value 2 standard deviations above the mean at the first measurement leads you to expect, at the second, about 1 standard deviation above the mean.
The noisier the measurement, the more marked the regression.
The most common traps
Rewards and punishments
An example told by Daniel Kahneman concerns flight instructors convinced that praising an excellent manoeuvre made the next one worse, while scolding a terrible one made it better. In reality, exceptional manoeuvres, in either direction, tended to be followed by more ordinary ones anyway. The effect attributed to praise and reproach was largely regression to the mean.
Interventions on the worst cases
You select the ten servers with the worst performance of the month, apply a new configuration, and the following month they improve. But selecting the worst also means selecting those with the worst luck: part of the improvement would have come anyway. The same applies to shops with the lowest sales, employees with the worst ratings, schools with the most disappointing results.
The «curse» of success
Those rewarded for an exceptional result often do not repeat it the following year. Not necessarily because of complacency or pressure: the exceptional year also contained a good dose of luck, and luck does not repeat on command.
Symptoms and remedies
People tend to seek a remedy when a complaint is at its peak. Since many complaints fluctuate, they are likely to improve in the following days whatever is done. This makes it very hard to evaluate a remedy on the basis of personal experience.
How to protect yourself
The control group
The most solid solution is to compare the treated group with a control group selected with the same criteria but not treated. If you take the twenty worst servers and apply the new configuration to ten of them chosen at random, both groups will regress towards the mean in the same way; the difference between the two measures the true effect of the intervention.
Repeated measurements before the intervention
Selecting on the basis of the average of several measurements, rather than a single one, reduces the weight of the random component and therefore the subsequent regression.
Do not select on the variable you want to improve
If possible, choose the units to be treated with a criterion different from the outcome measure, or at least compare with units selected in the same way.
Be wary of spectacular improvements
A very strong improvement after an intervention on extreme cases is exactly what regression to the mean predicts. Before celebrating it, ask how much you would have expected without doing anything.
An example to simulate
import numpy as np
rng = np.random.default_rng(3)
stable = rng.normal(100, 10, 1000) # true quality of each unit
month1 = stable + rng.normal(0, 10, 1000) # noisy measurement
month2 = stable + rng.normal(0, 10, 1000) # new measurement, no intervention
worst = month1.argsort()[-50:] # the 50 with the highest, i.e. worst, values
print(month1[worst].mean(), month2[worst].mean())
The worst units of the first month improve markedly in the second, without any intervention.
In short
- Every measurement combines a stable part and a random part; extremes often contain a lot of randomness.
- After an extreme value, the next measurement tends to be closer to the mean.
- The effect is stronger the lower the correlation between repeated measurements.
- Interventions on extreme cases seem to work even when they do nothing.
- A control group selected with the same criteria separates the real effect from regression.
