A help desk receives on average 3 tickets an hour. How likely is it that none arrive in an hour? And that 8 arrive? A server logs on average 2 errors a day: are 7 errors in one day normal variability or a signal worth investigating? To answer we need a model for counting events in an interval, and the simplest and most widespread is the Poisson distribution.
When it applies
The Poisson distribution describes the number of events occurring in an interval of time, space or volume when:
- events happen one at a time, not in groups;
- the numbers of events in non-overlapping intervals are independent;
- the average rate λ is constant over the interval considered.
Typical examples are calls to a switchboard, requests to a web service during a quiet minute, typos per page of a book, defects per metre of cable. A much-cited historical example is Ladislaus von Bortkiewicz's study of deaths from horse kicks in Prussian cavalry corps, which followed this distribution well.
The formula
If X ~ Poisson(λ), the probability of observing exactly k events is:
P(X = k) = e^(−λ) · λᵏ / k!
A remarkable property is that mean and variance coincide, both equal to λ.
The ticket example
With λ = 3 tickets an hour:
| k | P(X = k) |
|---|---|
| 0 | about 0.050 |
| 1 | about 0.149 |
| 2 | about 0.224 |
| 3 | about 0.224 |
| 4 | about 0.168 |
| 5 | about 0.101 |
| 6 or more | about 0.084 |
An hour without tickets happens about once in twenty. Six or more tickets in an hour, twice the average, occur in about 8% of hours: over an eight-hour working day it is quite likely to happen at least once. Planning staff on the average alone means being regularly overwhelmed.
The link with the binomial
The Poisson arises as the limit of the binomial when the number of trials n is very large, the probability p is very small and the product λ = n · p stays moderate. If a site receives 10,000 visits a day and each visit has a 0.02% probability of producing an error, the number of errors is binomial with n = 10,000 and p = 0.0002, but practically indistinguishable from a Poisson with λ = 2. The Poisson is handier because it needs only one parameter.
Changing the interval
If events follow a Poisson process, the rate rescales with the interval. With 3 tickets an hour, the number of tickets in half an hour is Poisson with λ = 1.5, and over an eight-hour day with λ = 24. Moreover, the sum of independent Poisson counts is still Poisson: if two queues receive on average 3 and 2 tickets an hour, together they receive on average 5, still Poisson distributed.
The time between one event and the next follows instead an exponential distribution with mean 1/λ: with 3 tickets an hour, the average wait between two tickets is 20 minutes.
Using it for alert thresholds
Back to the server with an average of 2 errors a day. The probability of observing 7 or more, if the rate really were 2, is about 0.0045, less than once every two hundred days. Observing it is therefore a good reason to investigate.
You can set a reasoned threshold: choose the value k such that P(X ≥ k) is below an acceptable false-alarm probability. In Python:
from scipy.stats import poisson
lam = 2
for k in range(4, 10):
print(k, poisson.sf(k - 1, lam)) # P(X >= k)
This way the alert does not fire on normal fluctuations, but flags truly unusual deviations.
When the model breaks down
Overdispersion
In real data the observed variance is often much larger than the mean. This is called overdispersion, and it indicates that the Poisson assumptions do not hold: the rate changes over time, or events arrive in clusters. An application's errors, for example, concentrate after a release or during a traffic spike. In these cases a model such as the negative binomial, or simply a different rate for each time slot, describes the data better.
Varying rate
Requests to a site do not have the same rate at three in the morning and at eleven in the morning. Applying a single Poisson to the whole day produces thresholds that are too low at night and too high during the day. It is better to estimate λ separately for each time slot.
Non-independent events
A user who gets an error often retries immediately, generating further errors. These events are not independent and inflate the counts. Before modelling, ask whether each event is truly a case of its own or the consequence of another.
A quick check
Before using the Poisson on a real count:
- compute the mean and variance of the counts per interval;
- if they are similar, the Poisson is a good starting point;
- if the variance is much larger, look for causes of heterogeneity or clustering;
- check that the rate is stable over the period considered.
In short
- The Poisson models the number of independent events in an interval with a constant rate.
- Mean and variance are both equal to λ.
- It is the limit of the binomial with many trials and a small probability.
- It lets you set alert thresholds based on a chosen false-alarm probability.
- A variance much larger than the mean signals that the model is not adequate.
