Variance and Standard Deviation
Maya and Jayden both have a $6.5$ average on their math quizzes. Maya's last four quizzes were $6, 7, 6, 7$. Jayden's were $3, 10, 3, 10$. The mean can't tell them apart, but they are very different students. To describe them you need a number that measures how far the data are from the mean.
What spread means
The simplest measure of spread is the range (maximum minus minimum). But the range only looks at the two extreme values and ignores everything in between. To use every data value, measure how far each one is from the mean, $x - \bar{x}$ (a deviation), and then average those distances.
Try it with Maya: the mean is $6.5$, so the deviations are $-0.5,\ +0.5,\ -0.5,\ +0.5$. Their sum is $0$.
In this page the data set is treated as the whole group, so we divide by $N$ (the number of data values).
Type a data set into the widget and watch how the spread changes. Try $6, 7, 6, 7$ and then $3, 10, 3, 10$.
Two students have the following grades in math: Student A: $6, 7, 6, 8, 7$ Student B: $4, 9, 5, 10, 6$ Without needing to do complex calculations, which of the two…
Two students have the following grades in math:
-
Student A: $6, 7, 6, 8, 7$
-
Student B: $4, 9, 5, 10, 6$
Without needing to do complex calculations, which of the two students do you think has a greater standard deviation? Confirm your answer by calculating the standard deviation for both.
Computing the variance
Follow the same three steps every time: find the mean, square each deviation, average the squares.
Data in a frequency table
When values repeat, use their frequencies $f_i$ as weights, and divide by the total $N = \sum f_i$:
$$\bar{x} = \frac{\sum f_i x_i}{N} \qquad \sigma^2 = \frac{\sum f_i (x_i - \bar{x})^2}{N}$$A bar chart is just a frequency table in disguise: read the height of each bar as the frequency of its value.
Calculate the variance and the standard deviation of the following set of data, corresponding to the ages of a group of five friends: $15, 16, 15, 17, 16$.
Calculate the variance and the standard deviation of the following set of data, corresponding to the ages of a group of five friends: $15, 16, 15, 17, 16$.
In a survey on the number of pets per household, the following results were obtained: $0$ pets (4 households), $1$ pet (5 households), $2$ pets (1 household). Calculate…
In a survey on the number of pets per household, the following results were obtained: $0$ pets (4 households), $1$ pet (5 households), $2$ pets (1 household). Calculate the variance of this distribution.
The following bar chart represents the number of books read during the summer by a group of students. From the graph, extract the frequencies corresponding to each value…
The following bar chart represents the number of books read during the summer by a group of students.
From the graph, extract the frequencies corresponding to each value and calculate the standard deviation of the number of books read.
The shortcut formula
When you already know $\sum x_i^2$ (from a table or a calculator), there is a faster way.
Why does it work? Expand the square and use $\sum x_i = N\bar{x}$:
$$\sum (x_i - \bar{x})^2 = \sum x_i^2 - 2\bar{x}\sum x_i + N\bar{x}^2 = \sum x_i^2 - 2N\bar{x}^2 + N\bar{x}^2 = \sum x_i^2 - N\bar{x}^2$$Divide by $N$ and you get the formula.
With a frequency table, $\sum x_i^2$ becomes $\sum f_i x_i^2$. For the siblings table: $\sum f_i x_i^2 = 0 + 4 + 8 + 9 = 21$, so $\sigma^2 = \dfrac{21}{10} - 1.1^2 = 2.1 - 1.21 = 0.89$. The same answer as before.
In a statistical study with $N = 40$ data values, it is known that the arithmetic mean is $\bar{x} = 5,2$ and that the sum of the squares of all the data values is…
In a statistical study with $N = 40$ data values, it is known that the arithmetic mean is $\bar{x} = 5,2$ and that the sum of the squares of all the data values is $\sum x_i^2 = 1250$. Using the short formula, calculate the variance and the standard deviation.
A set of data from a computing experiment consists exclusively of $20$ zeros and $30$ ones. Calculate the mean and the variance of this set of data.
A set of data from a computing experiment consists exclusively of $20$ zeros and $30$ ones. Calculate the mean and the variance of this set of data.
The monthly returns of an investment portfolio during the last semester have been: $-2\%, 3\%, 1\%, -1\%, 4\%, 1\%$. Calculate the variance and the standard deviation of…
The monthly returns of an investment portfolio during the last semester have been: $-2\%, 3\%, 1\%, -1\%, 4\%, 1\%$. Calculate the variance and the standard deviation of these returns expressed as a percentage.
Grouped data
Sometimes you only know how many values fall in each interval. Then use the midpoint of each class as its value $x_i$ and apply the same formulas with the frequencies. The result is an estimate, because we don't know the individual values inside each class.
The following table shows the time, in minutes, that the employees of a factory take to get to their workplace: $[0, 10)$: $12$ employees $[10, 20)$: $18$ employees…
The following table shows the time, in minutes, that the employees of a factory take to get to their workplace:
-
$[0, 10)$: $12$ employees
-
$[10, 20)$: $18$ employees
-
$[20, 30)$: $14$ employees
-
$[30, 40)$: $6$ employees
Calculate the variance and the standard deviation of the journey time using the class midpoints.
In the following grouped frequency table, two values are missing ($f_1$ and $f_2$): $[0, 20)$: $f_1$ $[20, 40)$: $10$ $[40, 60)$: $f_2$ $[60, 80)$: $5$ It is known that…
In the following grouped frequency table, two values are missing ($f_1$ and $f_2$):
-
$[0, 20)$: $f_1$
-
$[20, 40)$: $10$
-
$[40, 60)$: $f_2$
-
$[60, 80)$: $5$
It is known that the total number of data values is $N = 40$ and that the arithmetic mean is $\bar{x} = 37,5$. Calculate the values of the frequencies $f_1$ and $f_2$, and then find the variance of the complete distribution.
Shifting and scaling data
What happens to the mean and the spread when you change every data value in the same way?
Why? Adding $b$ moves the data and the mean together, so every deviation $x - \bar{x}$ stays the same. Multiplying by $a$ stretches every deviation by the factor $a$. Variance squares the deviations, so it gets the factor $a^2$. The square root then brings the factor back to $|a|$.
Standardizing
A very useful choice is $z = \dfrac{x - \bar{x}}{\sigma}$, which is $y = ax + b$ with $a = \dfrac{1}{\sigma}$ and $b = -\dfrac{\bar{x}}{\sigma}$. Its mean is $\dfrac{\bar{x} - \bar{x}}{\sigma} = 0$ and its standard deviation is $\dfrac{1}{\sigma}\cdot\sigma = 1$. Every standardized data set has mean $0$ and standard deviation $1$. A score of $85$ on a test with mean $70$ and $\sigma = 10$ becomes $z = \dfrac{85 - 70}{10} = 1.5$ ("one and a half standard deviations above the mean").
The standard deviation of the heights of a group of trees in a nursery is $12\text{ cm}$. If it is known that in a year's time all the trees will have grown exactly…
The standard deviation of the heights of a group of trees in a nursery is $12\text{ cm}$. If it is known that in a year's time all the trees will have grown exactly $15\text{ cm}$, what will be the new standard deviation of their heights? Justify your answer using the properties of measures of dispersion.
In a technology company, the variance of the monthly salaries of its employees is $40\,000\text{ \$}^2$. If the management decides to increase all salaries by $5\%…
In a technology company, the variance of the monthly salaries of its employees is $40\,000\text{ \$}^2$. If the management decides to increase all salaries by $5\%$ (that is, multiply them by $1,05$), what will be the new variance of the salaries?
A biological experiment has been carried out measuring the length of some leaves, obtaining a variable $X$ with mean $\bar{x} = 12\text{ cm}$ and standard deviation…
A biological experiment has been carried out measuring the length of some leaves, obtaining a variable $X$ with mean $\bar{x} = 12\text{ cm}$ and standard deviation $\sigma = 4\text{ cm}$. To facilitate calculations in a software program, a new "standardised" variable $Z = \frac{X - 12}{4}$ is defined. Prove numerically or by applying the properties of the mean and variance what the values of the mean and variance of the new variable $Z$ are.
Coefficient of variation
Which is more spread out: the heights of NBA players or the heights of gymnasts? Their standard deviations are in inches, but the two groups have very different means. A fair comparison needs a relative measure.
The $CV$ is also used in quality control: a process is acceptable when its $CV$ stays below a given limit. Only use it for data that are positive and measured from a true zero (heights, weights, prices), not for temperatures in °F.
In a 10th grade class, the mean grade on a history exam was $6,5$ points and the coefficient of variation is $20\%$. From this data, calculate the variance of the grades…
In a 10th grade class, the mean grade on a history exam was $6,5$ points and the coefficient of variation is $20\%$. From this data, calculate the variance of the grades of the class.
A coffee packaging machine must fill packets with a target weight of $250\text{ g}$. In a quality control check, a sample of $50$ packets was weighed, obtaining a mean…
A coffee packaging machine must fill packets with a target weight of $250\text{ g}$. In a quality control check, a sample of $50$ packets was weighed, obtaining a mean of $252\text{ g}$ and a sum of the squares of the weights of $\sum x_i^2 = 3\,175\,450$. Calculate the coefficient of variation and determine if the process is acceptable, knowing that the quality standard requires that $CV < 1,5\%$.
In a basketball team, the mean height of the players is $1,95\text{ m}$ with a standard deviation of $0,08\text{ m}$. On the other hand, in a rhythmic gymnastics team,…
In a basketball team, the mean height of the players is $1,95\text{ m}$ with a standard deviation of $0,08\text{ m}$. On the other hand, in a rhythmic gymnastics team, the mean height is $1,65\text{ m}$ with a standard deviation of $0,07\text{ m}$. Which team has a more dispersed height in relative terms? Justify your answer numerically.
Working backward from sums
Many problems give you a mean and a variance and ask about the original sums. Two identities do the job:
$$\sum x_i = N\bar{x} \qquad \sum x_i^2 = N\left(\sigma^2 + \bar{x}^2\right)$$The second one is just the shortcut formula solved for $\sum x_i^2$.
The same identities show that the mean of a transformed quantity is not the transformation of the mean.
The scores of four participants in a talent contest are $12, 15, x$ and $y$. It is known that the arithmetic mean of the four scores is $14$ and that $x = y$. Find the…
The scores of four participants in a talent contest are $12, 15, x$ and $y$. It is known that the arithmetic mean of the four scores is $14$ and that $x = y$. Find the value of the unknown scores and calculate the standard deviation of the set.
In a class of $25$ students, the mean grade on a physics exam is $6,2$ and the standard deviation is $1,4$. When reviewing the exams, the teacher realizes that the grade…
In a class of $25$ students, the mean grade on a physics exam is $6,2$ and the standard deviation is $1,4$. When reviewing the exams, the teacher realizes that the grade of one student, which was a $2$, had been recorded by mistake as that student did not take the exam. If this erroneous grade is removed from the data set, what will be the new mean and the new standard deviation of the remaining $24$ students?
A researcher measures the radius $r$ of a sample of $10$ metal spheres, obtaining a mean of $\bar{r} = 5\text{ cm}$ and a variance of $\sigma_r^2 = 2\text{ cm}^2$. They…
A researcher measures the radius $r$ of a sample of $10$ metal spheres, obtaining a mean of $\bar{r} = 5\text{ cm}$ and a variance of $\sigma_r^2 = 2\text{ cm}^2$. They need to know the mean surface area of these spheres to calculate the amount of paint needed. Knowing that the area of a sphere is $A = 4\pi r^2$, and using the definition of variance $\sigma_r^2 = \frac{\sum r_i^2}{N} - \bar{r}^2$, calculate the arithmetic mean of the surface areas of the $10$ spheres.
Combining two groups
If you know the size, mean and variance of two groups, you can find the variance of the combined group without the original data. Use $\sum x^2 = N(\sigma^2 + \bar{x}^2)$ for each group, add the two sums, and apply the shortcut formula to the total.
A language school has two B2 level groups. The morning group has $20$ students with a mean grade of $7$ and a variance of $1,5$. The afternoon group has $30$ students…
A language school has two B2 level groups. The morning group has $20$ students with a mean grade of $7$ and a variance of $1,5$. The afternoon group has $30$ students with the same mean grade of $7$, but with a variance of $2,5$. Calculate the overall variance of the $50$ B2 level students combined.
Two groups of students are combined to form a single class for a school trip. Group 1 has $15$ students with a mean height of $160\text{ cm}$ and a variance of…
Two groups of students are combined to form a single class for a school trip. Group 1 has $15$ students with a mean height of $160\text{ cm}$ and a variance of $25\text{ cm}^2$. Group 2 has $25$ students with a mean height of $168\text{ cm}$ and a variance of $16\text{ cm}^2$. Calculate the variance of the height of the whole class formed by the $40$ students.
There are two sets of data $A$ and $B$. Set $A$ has $N_A$ elements, mean $10$ and variance $4$. Set $B$ has $N_B$ elements, mean $10$ and variance $9$. If, when joining…
There are two sets of data $A$ and $B$. Set $A$ has $N_A$ elements, mean $10$ and variance $4$. Set $B$ has $N_B$ elements, mean $10$ and variance $9$. If, when joining both sets into a single group, the overall variance turns out to be $7$, determine the exact proportion between the number of elements in $A$ and in $B$ (that is, calculate the value of the fraction $\frac{N_A}{N_B}$).
Why the mean? Proofs and puzzles
Define the mean squared error from a constant $k$:
$$E(k) = \frac{1}{N}\sum (x_i - k)^2$$Where is it smallest? Drag the picture below: for the data $2, 4, 9$ (mean $5$) it is the parabola $E(x) = x^2 - 10x + 33.67$.
This is why variance squares the deviations and measures them from the mean. To prove formulas for $\{1, 2, \dots, n\}$, plug the known sums $\sum i$ and $\sum i^2$ into the shortcut formula and simplify. As a check, for $\{1, 2, 3\}$ the variance is $\dfrac{(1-2)^2 + 0 + 1^2}{3} = \dfrac{2}{3}$.
Puzzles: turn each condition into an equation
Show that the variance of the first $n$ natural numbers $\{1, 2, 3, \dots, n\}$ is given by the formula $\sigma^2 = \frac{n^2 - 1}{12}$. Apply this formula to quickly…
Show that the variance of the first $n$ natural numbers $\{1, 2, 3, \dots, n\}$ is given by the formula $\sigma^2 = \frac{n^2 - 1}{12}$. Apply this formula to quickly calculate the standard deviation of the integers from $1$ to $25$.
(Note: you can use the known sums $\sum_{i=1}^n i = \frac{n(n+1)}{2}$ and $\sum_{i=1}^n i^2 = \frac{n(n+1)(2n+1)}{6}$).
Find a set consisting of exactly $4$ positive integers (which can be repeated) that simultaneously satisfy the following three conditions: Their arithmetic mean is $5$.…
Find a set consisting of exactly $4$ positive integers (which can be repeated) that simultaneously satisfy the following three conditions:
-
Their arithmetic mean is $5$.
-
Their median is $4,5$.
-
Their variance is $4,5$.
Given a data distribution $x_1, x_2, \dots, x_N$, the function $E(k) = \frac{1}{N} \sum_{i=1}^N (x_i - k)^2$ is defined, which represents the mean squared error with…
Given a data distribution $x_1, x_2, \dots, x_N$, the function $E(k) = \frac{1}{N} \sum_{i=1}^N (x_i - k)^2$ is defined, which represents the mean squared error with respect to a constant value $k$. Expand the algebraic expression and show that the value of $k$ that minimizes this function is the arithmetic mean ($k = \bar{x}$), and that the minimum value reached by the function coincides exactly with the variance $\sigma^2$.
Back to the start: both Maya ($6, 7, 6, 7$) and Jayden ($3, 10, 3, 10$) have a mean of $6.5$. Maya's standard deviation is only $\sigma = 0.5$, while Jayden's is $\sigma = 3.5$. Now the number tells you what the mean could not.