Empirical Rule, Skewness and Outliers

Skewness: the shape of a distribution

A histogram shows how the data are spread out. Its shape tells you a lot.

• Symmetric: the left and right halves are mirror images. The mean and the median are about equal.

• Skewed to the right (positive skew): a long tail towards the higher values. A few large values pull the mean up, so the mean is greater than the median.

• Skewed to the left (negative skew): a long tail towards the lower values. A few small values pull the mean down, so the mean is less than the median.

The skew is named after the tail, not after where the peak is.

Now return to the Dublin houses. The mean (€420,000) is bigger than the median (€365,000), so the distribution is skewed to the right: a few very expensive houses pull the mean up. We come back to which average to quote in the last section.

The empirical rule

Many real measurements, such as heights, egg masses or journey times, give a symmetric, bell-shaped histogram. We call this an approximately normal distribution. For these data, the empirical rule tells you how much of the data lies close to the mean $\bar{x}$ (the standard deviation is $\sigma$).

Because the curve is symmetric, you can split the percentages into the tails:

• Within $1\sigma$: $68\%$, so $100 - 68 = 32\%$ is outside, i.e. $16\%$ in each tail.

• Within $2\sigma$: $95\%$, so $5\%$ is outside, i.e. $2.5\%$ in each tail.

• Within $3\sigma$: $99.7\%$, so only $0.3\%$ is outside, i.e. $0.15\%$ in each tail.

Quartiles, IQR and outliers

The standard deviation measures spread around the mean. The interquartile range measures the spread of the middle half of the data and is not affected by extreme values.

An outlier is a value that is unusually far from the rest of the data. A common test says a value is an outlier if it is:

• below $Q_1 - 1.5 \times \text{IQR}$, or

• above $Q_3 + 1.5 \times \text{IQR}$.

The reason is that the IQR describes the usual spread, so a value more than one and a half times that spread beyond the quartiles is far outside it.

How outliers affect the mean and median

Try the data set below. Change the last value and watch what happens to the mean and median.

The mean uses every value, so one very large or very small value drags it towards that value. The median only depends on the middle position, so it hardly moves.

For the hours-online data in the last section, the mean is $\frac{111}{10} = 11.1$ and the median is $8.5$. Remove the outlier $40$ and the mean falls to $\frac{71}{9} \approx 7.9$, while the median only changes from $8.5$ to $8$. The mean is bigger than the median because of the large outlier, which is a sign of right skew.

Back to the Dublin houses: a few very expensive properties pull the mean up to €420,000. The median of €365,000 is the better description of a typical house.

To compare two data sets, use a measure of centre and a measure of spread together: for example the mean and standard deviation for symmetric data, or the median and IQR for skewed data. A larger standard deviation means the data are more spread out.

Summary