Correlation tells you how strongly two quantities are linearly related. Regression goes one step further: it describes how they are related, with an equation that lets you estimate one quantity from the other. If a server's CPU load increases with the number of connected users, regression answers the practical question: by how much does the load increase for every hundred extra users?
The model
Simple linear regression describes the relationship between an explanatory variable x and a response variable y with a straight line:
y = a + b · x + error
- b is the slope: how much y changes on average when x increases by one unit;
- a is the intercept: the predicted value of y when x is zero;
- the error represents everything the line does not explain: measurement noise, factors not considered, natural variability.
The least squares method
Among all possible lines, we choose the one that minimises the sum of squared residuals, where a point's residual is the difference between the observed value and the value predicted by the line. The resulting formulas are simple:
- b = Σ (x − x̄)(y − ȳ) / Σ (x − x̄)²
- a = ȳ − b · x̄
The line always passes through the mean point (x̄, ȳ).
An example worked by hand
We measure CPU load with different numbers of users, expressed in hundreds:
| Users (hundreds) | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| CPU (%) | 22 | 30 | 41 | 48 | 59 |
The means are x̄ = 3 and ȳ = 40.
- Numerator: (−2)(−18) + (−1)(−10) + 0 · 1 + 1 · 8 + 2 · 19 = 36 + 10 + 0 + 8 + 38 = 92.
- Denominator: 4 + 1 + 0 + 1 + 4 = 10.
- Slope: b = 92 / 10 = 9.2.
- Intercept: a = 40 − 9.2 · 3 = 12.4.
The line is CPU = 12.4 + 9.2 · users. Every hundred extra users add on average about 9.2 percentage points of load. The intercept of 12.4 can be read as the system's baseline load with no users, but only if the model remains valid down to zero, which needs to be checked.
How much the line explains: R²
The coefficient of determination R² compares residual variability with total variability:
R² = 1 − Σ residuals² / Σ (y − ȳ)²
In the example the predicted values are 21.6, 30.8, 40, 49.2, 58.4, and the residuals 0.4, −0.8, 1, −1.2, 0.6. The sum of their squares is 3.6, while the total variability is 850. So R² ≈ 1 − 3.6/850 ≈ 0.996. The line describes these data almost perfectly.
In simple regression R² equals the square of Pearson's correlation coefficient. A high R², however, does not guarantee that the model is correct, and a low R² does not mean the relationship is useless: even a small but stable effect can be important.
Reading the residuals
Residuals are the most informative part of a regression. A plot of residuals against predicted values should show a shapeless cloud centred on zero. Some patterns signal problems:
- a curve: the relationship is not linear, and a transformation or a different model is needed;
- a widening fan: variability grows with x, and standard confidence intervals are not reliable;
- isolated points far away: outliers that can shift the line;
- trends over time: if the data are collected in sequence, correlated residuals indicate that observations are not independent.
The model's assumptions
To use regression not only as a description but also for inference, that is, confidence intervals and tests on the slope, some assumptions are needed:
- linearity of the mean relationship;
- independence of the observations;
- constant variance of the errors;
- approximately normal errors, important mainly with small samples.
The danger of extrapolation
The line was estimated between 100 and 500 users. Using it for 1,000 users would give a predicted load of 12.4 + 9.2 · 10 = 104.4%, an impossible value. In reality, as saturation approaches, behaviour changes: queues, slowdowns, errors. A model is valid only within the range of the data on which it was built, and outside that range it can be very wrong.
With Python
import numpy as np
x = np.array([1, 2, 3, 4, 5])
y = np.array([22, 30, 41, 48, 59])
b, a = np.polyfit(x, y, 1)
residuals = y - (a + b * x)
r2 = 1 - (residuals ** 2).sum() / ((y - y.mean()) ** 2).sum()
print(a, b, r2)
For confidence intervals, tests and full diagnostics it is better to use dedicated libraries such as statsmodels.
Regression does not mean causation
An estimated slope tells you how y varies on average as x varies in the observed data. It does not say that changing x will produce that change in y. If x and y both depend on a third factor, the regression faithfully reproduces the association, but not the effect of an intervention.
In short
- Linear regression describes the average relationship between two quantities with a straight line.
- Least squares chooses the line that minimises the sum of squared residuals.
- R² measures the share of variability explained, but does not guarantee the model is correct.
- The residual plot reveals nonlinearity, non-constant variance and outliers.
- The model holds only within the range of the data and does not by itself prove a causal link.
