Simpson's paradox: when aggregated data reverse

Two technical support teams each handle 500 tickets in a month. Team A resolves 42% at first contact, team B 73%. It seems obvious which of the two works better. Then it turns out that team A is better than B on both simple tickets and complex ones. It is not a calculation error: it is Simpson's paradox, one of the most instructive phenomena in statistics.

The example in detail

Let us split the tickets into simple and complex:

Team A has a higher resolution rate in both categories, yet a much lower overall rate. How is this possible?

The mechanism

The key is the composition of the groups. Team A receives mostly complex tickets: 400 out of 500. Team B receives mostly simple tickets: 400 out of 500. Each team's overall rate is an average of the per-category rates, weighted by the share of tickets in each category.

• A's 42% is dominated by the 30% on complex tickets, which make up 80% of its work.
• B's 73% is dominated by the 85% on simple tickets, which make up 80% of its work.

Ticket difficulty is a confounding variable: it influences both the assignment to a team and the probability of resolution. Comparing totals means comparing mainly the type of work received, not skill.

Famous examples

The paradox appears in many real contexts. One of the most cited cases concerns graduate admissions at the University of California, Berkeley in 1973: overall, women were admitted at a lower rate than men, but when individual departments were analysed the disparity against women largely disappeared. The explanation was that women applied more often to departments with low admission rates for everyone.

The same pattern is found in medicine, when a treatment is given more often to the most severely ill patients, and in the analysis of digital products, when a feature is used mainly by a particular type of user.

Which figure is the right one?

The most important answer is also the most uncomfortable: it depends on the causal structure, not on the numbers. Statistics alone cannot say whether to use the aggregated data or the subgroup data.

When to use subgroups

If the variable that splits the groups is a common cause of both the «treatment» and the outcome, like ticket difficulty, you must compare within the subgroups. In the example, difficulty precedes assignment and influences the outcome: team A really is better.

When to use the aggregate

If instead the variable that splits the groups is a consequence of the treatment, stratifying is wrong. Suppose a new triage process causes more tickets to be classified as complex and handled with more care. Then the «complex» category is an effect of the process, and comparing subgroups would hide precisely the effect we want to measure. In that case the aggregated figure is the correct one.

The question to ask

Before choosing, ask yourself: does the variable I am using to split the data come before the treatment and influence it, or does it come after and is influenced by it? Drawing a small diagram with causal arrows helps a great deal.

How to notice it

The paradox does not announce itself: you have to go looking for it.

1. Disaggregate the data by the variables that plausibly influence the outcome: customer type, device, region, period.
2. Compare compositions: if the groups being compared have very different proportions of subcategories, totals can mislead.
3. Look at the weights: an overall rate is always a weighted average, and weights can matter more than rates.
4. Randomise when you can: in a randomised experiment the composition of the groups is on average the same, and the paradox cannot arise from a confounder.

With pandas, subgroup analysis is immediate:

import pandas as pd

df = pd.read_csv("tickets.csv")  # columns: team, type, resolved (0/1)
print(df.groupby("team")["resolved"].mean())
print(df.groupby(["type", "team"])["resolved"].mean().unstack())
print(pd.crosstab(df["team"], df["type"], normalize="index"))

The last line shows the composition of each group, which often explains the paradox on its own.

The link with correlation

Simpson's paradox also applies to relationships between continuous variables: two quantities can be negatively correlated within every group and positively correlated overall. It is another reason why a correlation computed on heterogeneous data should be treated with extreme caution.

In short

• A group can be better in every subgroup and worse in the total.
• The paradox arises from different group compositions and a confounding variable.
• Overall rates are weighted averages, and the weights can dominate the result.
• Choosing between aggregated and stratified data requires reasoning about causal structure.
• Disaggregating the data and comparing compositions is how you discover it.