Regression Line (Line of Best Fit)

Scatter plots and the line of best fit

In a scatter plot, each dot is one student. The $x$-value is the hours studied and the $y$-value is the score. When the dots drift upward from left to right, more hours go with higher scores. We call this a positive association.

A line of best fit is a line drawn through the cloud of dots that follows its trend. Once you have it, you can predict $y$ for a new $x$. Go up from $x$ to the line, then straight across to the $y$-axis.

The best-fit line used in statistics is the regression line (more precisely, the least-squares regression line). The next sections explain exactly how it is chosen. First, let's get comfortable using one.

The line below has two sliders. Move them and watch how $m$ and $b$ change the line.

The regression line formula

Drawing by eye is not precise: two people draw two different lines. Statistics gives one exact line from three numbers.

Recall the mean of each variable, $\bar{x}$ and $\bar{y}$. Together they give the mean point $(\bar{x}, \bar{y})$, also called the center of gravity of the data. Recall also:

• the variance of $X$: $\sigma_x^2 = \dfrac{1}{n}\sum (x - \bar{x})^2$, which measures how spread out the $x$-values are;

• the covariance: $\sigma_{xy} = \dfrac{1}{n}\sum (x - \bar{x})(y - \bar{y})$, which is positive when $x$ and $y$ rise together and negative when one rises while the other falls.

Why this slope? The covariance measures how much $x$ and $y$ move together. Dividing by the variance of $x$ turns that into "how much $y$ moves per unit of $x$". Why the mean point? The line is built in point-slope form around that point, so it must pass through it.

Because the line always passes through the mean point, you can use it in two ways. If the line is $y = 2x + 1$ and $\bar{x} = 3$, then $\bar{y} = 2(3) + 1 = 7$. If a candidate line misses the mean point, it is not the regression line. Hitting the mean point is necessary but not enough: many lines pass through that point.

Computing the line from a data table

When you are given the data points, you compute the means, the deviations from the means, and two sums:

Since $\sigma_{xy} = S_{xy}/n$ and $\sigma_x^2 = S_{xx}/n$, the $n$'s cancel and the slope is simply

This is the same line a calculator gives, even though calculators sometimes divide by $n-1$ in the covariance and variance.

Residuals: how far off is the line?

The line does not hit every dot. The vertical gap between a dot and the line is the residual.

The regression line is the line that makes the sum of the squares of the residuals as small as possible. That is why it is also called the least-squares line.

Predicting with care

A prediction is only as good as the data behind it. Keep three ideas in mind.

1. Interpolation vs. extrapolation. Predicting for an $x$ inside the range of the data is interpolation, which is usually safe. Predicting outside that range is extrapolation, which is risky. Nothing guarantees the trend continues.

2. Do not "flip" the line. The line of $Y$ on $X$ minimizes errors in $Y$. To estimate $X$ from a known $Y$, you cannot just solve the equation for $x$. The rigorous way is to compute a different line, the regression line of $X$ on $Y$ (see the section on the two regression lines).

3. Interpret slope and intercept in context. The intercept is the predicted $y$ when $x = 0$. If $x = 0$ is far outside the data, it may mean nothing.

The correlation coefficient $r$

How well do the dots hug the line? The correlation coefficient gives a number for that.

Why is the sign of $r$ the same as the sign of the slope? The slope is $\sigma_{xy}/\sigma_x^2$ and $r = \sigma_{xy}/(\sigma_x\sigma_y)$. Variances and standard deviations are positive, so both have the same sign as the covariance $\sigma_{xy}$. This also means the sign of the slope always matches the sign of the covariance.

That tells you which graph supports a reliable prediction. With a tight cloud around the line (large $|r|$), the residuals are small and predictions are trustworthy. With a loose cloud (small $|r|$), the line is only a rough guide.

From $r$ to the slope, and back

Since $\sigma_{xy} = r\,\sigma_x\sigma_y$, the slope can be written with $r$ instead of the covariance:

The slope is the correlation times the ratio of the spreads. Squares are for variances ($\sigma^2$), plain $\sigma$'s are for standard deviations. Use whichever the problem gives you.

You can also go backward and find $r$ from a given line: $r = m\,\dfrac{\sigma_x}{\sigma_y}$.

Using a calculator or spreadsheet

In practice, nobody does $S_{xy}$ by hand with large data sets. A graphing calculator or Desmos does it in seconds.

Try this made-up data set: days practiced per week $x$ and correct answers out of $20$ on a quiz $y$: $(1, 12)$, $(2, 13)$, $(3, 15)$, $(4, 14)$, $(5, 16)$.

• TI-84: press STAT → Edit and enter $x$ in L1 and $y$ in L2. Turn Diagnostics On (in the CATALOG), then press STAT → CALC → LinReg(ax+b) L1, L2. You get $a = 0.9$, $b = 11.3$ and $r = 0.9$.

• Desmos: make a table with columns $x_1$ and $y_1$, then type y_1 ~ m x_1 + b. Desmos reports the same $m$ and $b$ and also shows the residuals.

By hand we can confirm: $\bar{x} = 3$, $\bar{y} = 14$, $S_{xy} = 9$, $S_{xx} = 10$, so $m = 0.9$ and $b = 14 - 0.9(3) = 11.3$. The line is $y = 0.9x + 11.3$. For $4$ practice days the line predicts $0.9(4) + 11.3 = 14.9$ correct answers. That is an estimate, not a promise.

The two regression lines

So far we predicted $Y$ from $X$. We can also predict $X$ from $Y$. Swapping the roles gives the regression line of $X$ on $Y$:

Both lines pass through the mean point $(\bar{x}, \bar{y})$. If you are given both equations, solving them as a system (substitution) gives that point. The two slopes (written as $y = m_1 x + \cdots$ and $x = m_2 y + \cdots$) multiply to give $r^2$:

Why $r = \pm 1$ makes the two lines the same. When $r^2 = 1$, the product of the slopes is $1$. The line of $X$ on $Y$ is $x - \bar{x} = m_2 (y - \bar{y})$. Solving for $y$ gives $y - \bar{y} = \dfrac{1}{m_2}(x - \bar{x})$, whose slope is $\dfrac{1}{m_2}$. Because $m_1 m_2 = 1$, we have $\dfrac{1}{m_2} = m_1$. So both lines have the same slope and the same point $(\bar{x}, \bar{y})$, which makes them one line. This is the case where all the dots lie exactly on that line.

Changing the data: shifts and rescaling

What happens to the line when you change units or add bonus points? Just substitute.

Why? If every $y$ goes up by $c$, every predicted value goes up by $c$ too, so the whole line shifts up. If $x$ is measured in units that are $k$ times as large, a single unit of $z$ is a bigger step, so $y$ changes by $k$ times as much per unit. We then divide by $k$.

For example, if Mr. Lee adds $5$ bonus points to every quiz score and the line was $y = 0.8x + 10$, the new line is $y = 0.8x + 15$.

Adding or removing a point at the mean

Some points have no influence on the slope. To see why, use sums. The $n$'s cancel in $m = S_{xy}/S_{xx}$, and there is a handy fact: because $\sum (x - \bar{x}) = 0$,

If the point added or removed is exactly $(\bar{x}, \bar{y})$, then nothing at all changes: the mean point stays the same and so does the line.

Two special lines

Standardized variables. Suppose both variables have been standardized: $\bar{x} = \bar{y} = 0$ and $\sigma_x = \sigma_y = 1$. Then $\sigma_{xy} = r\sigma_x\sigma_y = r$, and the regression line $y - \bar{y} = r\dfrac{\sigma_y}{\sigma_x}(x - \bar{x})$ simplifies to

If the variables are uncorrelated ($r = 0$), the line is $y = 0$: the horizontal $x$-axis.

A horizontal regression line. The slope is $\dfrac{\sigma_{xy}}{\sigma_x^2}$. If the line is horizontal, the slope is $0$, so $\sigma_{xy} = 0$ (since $\sigma_x^2 > 0$). Then $r = 0$ and the line is $y = \bar{y}$. Knowing $x$ does not improve the prediction of $y$: we always predict the mean. We conclude there is no linear relationship, though a curved one could still exist.

Correlation is not causation

A value of $r$ near $1$ says two variables move together very regularly. It does not say why.

On hot days more people buy ice cream, and more people get sunburned. The two have a high correlation, but ice cream does not burn anyone's skin. A third factor, the lurking variable (weather), drives both.

Back to Jayden: his teacher's line predicts $4(7) + 56 = 84$ for 7 hours of study. The prediction is reasonable because $7$ hours lies inside the data range and the dots hug the line. It is an estimate, not a guarantee. A strong correlation here does not prove that studying alone causes the score; things like prior knowledge and sleep matter too.

Summary