Pearson and Spearman correlation: reading it without errors

«The two quantities have a correlation of 0.8.» It is one of the most common sentences in any data report, and also one of the easiest to over-interpret. The correlation coefficient is a useful number, but it compresses an entire cloud of points into a single value, and in this compression the decisive information can be lost.

Covariance as a starting point

Consider pairs of values (x, y), for example the number of connected users and a server's CPU load, measured every minute. Covariance measures whether the two quantities tend to be above or below their respective means together:

• if when x is above its mean y also tends to be, the covariance is positive;
• if when x is above its mean y tends to be below, it is negative;
• if there is no tendency, it is close to zero.

The problem with covariance is that it depends on the units of measurement: expressing load as a percentage or as a fraction changes its value. This is why it is normalised.

Pearson's coefficient

Pearson's correlation coefficient is the covariance divided by the product of the two standard deviations:

r = Cov(x, y) / (σₓ · σᵧ)

The result always lies between −1 and +1:

• r = +1: the points lie exactly on an increasing straight line;
• r = −1: exactly on a decreasing straight line;
• r = 0: no linear relationship.

The square r² indicates the share of one quantity's variability «explained» linearly by the other: with r = 0.8, r² = 0.64, that is 64%. Even a correlation that sounds high therefore leaves a third of the variability unexplained.

Anscombe's quartet

In 1973 the statistician Francis Anscombe built four small data sets with practically identical summary statistics: same means, same variances, same regression line and a correlation of about 0.82 in all four. Yet, plotted on a graph, they are completely different:

1. the first is a cloud of points with a linear trend and some noise, the «normal» case;
2. the second is a perfectly smooth but nonlinear curve;
3. the third is an almost perfect straight line with a single outlier that lowers the correlation;
4. the fourth has all points at the same x value except one, which on its own creates the entire correlation.

The lesson is simple and fundamental: before computing a correlation, look at the scatter plot.

The main traps

Nonlinear relationships

Pearson measures only linear relationships. If y = x² and x runs from −10 to +10, the relationship is perfectly deterministic, yet r equals 0. A correlation close to zero does not mean the two quantities are independent.

Outliers

A single point far from the others can create a strong correlation where there is none, or cancel one that exists. With small samples the effect is particularly pronounced.

Restricted range

If only part of the possible range is observed, the correlation shrinks. Among servers already running at 90-95% load, the relationship between users and response times may look weak because the variability of the load is minimal; widening the view to all load levels makes the relationship evident.

Mixed groups

Merging different groups can create correlations that exist in none of the groups, or reverse them. This is the territory of Simpson's paradox, which deserves separate treatment.

Spearman's correlation

Spearman's correlation is computed by applying Pearson's formula not to the values but to their ranks, that is, their positions in the ordering. It therefore measures whether one quantity tends to increase when the other does, even if not linearly: this is called a monotonic relationship.

It has two practical advantages:

• it is robust to extreme values, because a huge value simply becomes «first in the ranking»;
• it captures nonlinear monotonic relationships, such as exponential growth or saturation.

If y = eˣ, Spearman equals exactly 1, while Pearson is less than 1.

from scipy.stats import pearsonr, spearmanr

x = [1, 2, 3, 4, 5, 6, 7, 8]
y = [2, 4, 8, 16, 32, 64, 128, 256]
print(pearsonr(x, y)[0])    # less than 1
print(spearmanr(x, y)[0])   # exactly 1

Correlation and significance

With few points, even strong correlations can arise by chance: with 5 pairs of random data it is not rare to get |r| > 0.7. With millions of points, by contrast, tiny correlations come out statistically significant. It is always worth reporting r together with the number of observations and, if possible, a confidence interval.

The question that remains open

Even a strong, well-measured and significant correlation does not say why two quantities vary together. There may be a causal link, in either direction, or a common cause, or a selection of the data. Correlation is a clue to investigate, not a conclusion.

In short

• Pearson's coefficient measures the strength of the linear relationship between −1 and +1.
• r² indicates the share of variability explained linearly, often less than it seems.
• Anscombe's quartet shows that identical statistics can hide very different data.
• Spearman works on ranks: it is robust and captures nonlinear monotonic relationships.
• A zero correlation does not imply independence, and a strong one does not imply causation.