Hypothesis Tests
Jayden, the starting guard, says he makes $70\%$ of his free throws. In Friday's game he makes only $28$ of $50$ shots, which is $56\%$. Is he just having an off night, or is $70\%$ too high a claim? A hypothesis test (also called a significance test) gives a fair, standard way to decide. We will come back to Jayden at the end of the page.
Stating the hypotheses
Every test starts with two competing statements about a parameter, a number that describes a whole population (not just your sample).
The null hypothesis $H_0$ is the "nothing special is going on" claim. It says the parameter equals one specific value. The alternative hypothesis $H_a$ is what we suspect and want evidence for. It says the parameter is less than, greater than, or different from that value.
Think of a courtroom: the defendant is presumed innocent ($H_0$) until the evidence is strong enough to convict ($H_a$). We never "prove innocence." We only decide whether the evidence is strong enough to reject it.
The direction of $H_a$ comes from the question being asked, before you look at the data. The shaded tails above show where the sample result would have to land to count as evidence for $H_a$.
| Situation | Parameter | Null hypothesis |
|---|---|---|
| One proportion | $p$ | $H_0: p = p_0$ |
| One mean | $\mu$ | $H_0: \mu = \mu_0$ |
| Paired data | $\mu_d$ (mean of the differences) | $H_0: \mu_d = 0$ |
| Two proportions | $p_1 - p_2$ | $H_0: p_1 = p_2$ |
| Two means | $\mu_1 - \mu_2$ | $H_0: \mu_1 = \mu_2$ |
A state senator claims that more than $60\%$ of her constituents support the new highway infrastructure bill. A local news station conducts a random phone poll of…
A state senator claims that more than $60\%$ of her constituents support the new highway infrastructure bill. A local news station conducts a random phone poll of registered voters in her district to test this claim. Let $p$ represent the true proportion of constituents who support the bill. State the appropriate null and alternative hypotheses for this test.
Test statistics, P-values, and decisions
The logic of every test is the same: assume $H_0$ is true, then ask how surprising the sample result would be.
First, standardize the result with a test statistic:
$$\text{test statistic} = \frac{\text{statistic} - \text{parameter value under } H_0}{\text{standard deviation (or standard error) of the statistic}}$$It counts how many standard deviations the sample result is from what $H_0$ predicts. Then the P-value measures how surprising that is.
A small P-value means the data would be rare if $H_0$ were true. That is evidence against $H_0$. For a two-sided test, count both tails (double the one-tail area).
The significance level $\alpha$ (usually $0.05$) is the cutoff you choose in advance for "rare enough."
Standard normal cutoffs are worth memorizing, because they let you judge a $z$ statistic at a glance:
| Test | $\alpha = 0.05$ | $\alpha = 0.01$ |
|---|---|---|
| One-sided | $z = \pm 1.645$ | $z = \pm 2.326$ |
| Two-sided | $z = \pm 1.96$ | $z = \pm 2.576$ |
On the AP exam, answers follow four steps: State (hypotheses, parameter, $\alpha$), Plan (name the test and check conditions), Do (statistic and P-value), Conclude (decision in context).
A quality control manager at a cereal factory conducts a hypothesis test to determine if the mean weight of their standard cereal boxes is less than the advertised $16$…
A quality control manager at a cereal factory conducts a hypothesis test to determine if the mean weight of their standard cereal boxes is less than the advertised $16$ ounces. The test results in a P-value of $0.034$. Using a significance level of $\alpha = 0.05$, state the decision (reject or fail to reject the null hypothesis) and provide the conclusion in the context of the problem.
A sociologist uses statistical software to compare the mean annual salaries (in thousands of dollars) of male and female software engineers with $5$ years of experience.…
A sociologist uses statistical software to compare the mean annual salaries (in thousands of dollars) of male and female software engineers with $5$ years of experience. The software output for a two-sample t-test shows:
T-Value = 2.45, P-Value = 0.018, DF = 48.
Assuming the test is two-sided and the chosen significance level is $\alpha = 0.01$, what decision should the researcher make regarding the null hypothesis, and what is the conclusion in context?
The graph below shows the standard normal distribution for a two-sided hypothesis test regarding a population proportion. The calculated test statistic from the sample…
The graph below shows the standard normal distribution for a two-sided hypothesis test regarding a population proportion. The calculated test statistic from the sample data is $z = 2.15$.
Explain what the shaded red regions represent in terms of the P-value. Based on standard normal probabilities, state whether the null hypothesis should be rejected at the $\alpha = 0.05$ significance level.
Tests for proportions
One proportion: the $z$ test
You have one random sample and a claimed proportion $p_0$. The test statistic is
$$z = \frac{\hat{p} - p_0}{\sqrt{\dfrac{p_0(1 - p_0)}{n}}}$$The denominator is the standard deviation of $\hat{p}$ if $H_0$ is true, which is why it uses $p_0$, not $\hat{p}$. The P-value comes from the standard normal curve.
Two proportions: pooling
To compare two groups, $H_0: p_1 = p_2$. If $H_0$ is true, both groups share one common proportion, so the best estimate of it combines the two samples. This is the pooled (combined) proportion:
$$\hat{p}_c = \frac{x_1 + x_2}{n_1 + n_2}, \qquad z = \frac{\hat{p}_1 - \hat{p}_2}{\sqrt{\hat{p}_c(1 - \hat{p}_c)\left(\dfrac{1}{n_1} + \dfrac{1}{n_2}\right)}}$$Conditions: both samples random (or randomized experiment), each sample under $10\%$ of its population, and large counts: $n_1\hat{p}_c$, $n_1(1 - \hat{p}_c)$, $n_2\hat{p}_c$, $n_2(1 - \hat{p}_c)$ all at least $10$. (Some texts check the actual counts of successes and failures in each group.)
Working backward
Test statistics are formulas, so you can solve them for a missing piece. With equal sample sizes $n$, the standard error simplifies to $\sqrt{2\hat{p}_c(1 - \hat{p}_c)/n}$.
A nationwide fast-food chain claims that $20\%$ of their drive-thru customers order a vegetarian meal. A skeptical franchise owner takes a random sample of $150$…
A nationwide fast-food chain claims that $20\%$ of their drive-thru customers order a vegetarian meal. A skeptical franchise owner takes a random sample of $150$ drive-thru customers and finds that $36$ of them ordered a vegetarian meal. Calculate the test statistic ($z$) that should be used to test the chain's claim.
A commercial airline claims that less than $8\%$ of its domestic flights are delayed by more than $30$ minutes. A consumer rights group takes a random sample of $400$…
A commercial airline claims that less than $8\%$ of its domestic flights are delayed by more than $30$ minutes. A consumer rights group takes a random sample of $400$ domestic flights from this airline and finds that $26$ of them were delayed by more than $30$ minutes. Check if the conditions for inference are met, and if so, calculate the appropriate test statistic to evaluate the airline's claim.
A tech company is running an A/B test on two versions of a checkout page to see which yields a higher conversion rate. Version A is shown to $n$ users and Version B is…
A tech company is running an A/B test on two versions of a checkout page to see which yields a higher conversion rate. Version A is shown to $n$ users and Version B is shown to a separate group of $n$ users (equal sample sizes). The pooled sample proportion of users who completed a purchase across both groups is $\hat{p}_c = 0.16$.
The test statistic for $H_0: p_A = p_B$ vs $H_a: p_A \neq p_B$ is calculated to be $z = -2.50$. If the observed difference in sample proportions is $\hat{p}_A - \hat{p}_B = -0.04$, calculate the sample size $n$ for each group. Show the algebraic steps used to find $n$.
One mean and paired data: the $t$ test
With means, we almost never know the population standard deviation $\sigma$, so we use the sample standard deviation $s$. That extra uncertainty makes the statistic follow a $t$ distribution with $df = n - 1$ degrees of freedom.
$$t = \frac{\bar{x} - \mu_0}{s/\sqrt{n}}, \qquad df = n - 1$$Find the P-value with a $t$ table (bracket it between two columns) or a calculator: tcdf(lower, upper, df).
Paired data
Sometimes each subject is measured twice (before/after, left/right), or subjects are matched in pairs. Then the data are paired. Compute the difference within each pair and run a one-sample $t$ test on those differences, with $\mu_0 = 0$ and $n$ = number of pairs. The parameter is $\mu_d$, the true mean difference.
Working backward works here too. If a problem gives $t$, $s$, $n$, and $\mu_0$, solve for $\bar{x}$:
$$\bar{x} = \mu_0 + t \cdot \frac{s}{\sqrt{n}}$$A high school teacher wants to test if a new intensive SAT prep book improves students' math scores. She records the math scores of $15$ students before they use the…
A high school teacher wants to test if a new intensive SAT prep book improves students' math scores. She records the math scores of $15$ students before they use the book and their scores on a similar test after completing the book. Let $\mu_d$ be the true mean difference (After $-$ Before) in SAT math scores. State the appropriate null and alternative hypotheses to test if the book improves scores.
A popular coffee shop chain claims their large iced coffee contains an average of $24$ fluid ounces. A skeptical customer believes they are being shortchanged and takes…
A popular coffee shop chain claims their large iced coffee contains an average of $24$ fluid ounces. A skeptical customer believes they are being shortchanged and takes a random sample of $12$ large iced coffees. The sample yields a mean of $23.2$ fluid ounces with a standard deviation of $0.9$ fluid ounces. Assuming the volume of coffee is normally distributed, calculate the test statistic and identify the degrees of freedom for this hypothesis test.
A school district conducted a one-sample t-test to determine if the mean reading score of its 5th-grade students was significantly higher than the state average of…
A school district conducted a one-sample t-test to determine if the mean reading score of its 5th-grade students was significantly higher than the state average of $210$. The hypotheses were $H_0: \mu = 210$ vs $H_a: \mu > 210$. A random sample of $25$ students yielded a test statistic of $t = 2.064$. If the sample standard deviation was exactly $15$ points, what was the sample mean reading score ($\bar{x}$) of these $25$ students?
Comparing two means
When you have two independent groups (no pairing between them), use the two-sample $t$ test for $H_0: \mu_1 = \mu_2$:
$$t = \frac{\bar{x}_1 - \bar{x}_2}{\sqrt{\dfrac{s_1^2}{n_1} + \dfrac{s_2^2}{n_2}}}$$Calculators and software compute a (fractional) number of degrees of freedom. For work by hand, use the conservative value $df = \min(n_1 - 1, \ n_2 - 1)$. That makes the P-value a bit larger, so it is the safe choice.
Paired or two samples?
Ask: is each value in one group naturally linked to one value in the other group? If the same subject is measured twice (or subjects are matched), the data are paired. If the groups are separate, use two samples.
Pairing is powerful. In a study of a new track shoe, Design 1 (different athletes in each group) mixes the effect of the shoe with the natural differences in speed between athletes. Design 2 (each athlete runs in both shoes, order randomized) subtracts each athlete's own baseline, so athlete-to-athlete variability disappears from the differences. A smaller standard error makes a real effect easier to detect.
A physical therapist wants to know if there is a significant difference in flexibility between the left and right legs of professional soccer players. She measures the…
A physical therapist wants to know if there is a significant difference in flexibility between the left and right legs of professional soccer players. She measures the maximum extension angle of both the left and right legs for a random sample of $20$ players. To analyze this data, should the therapist use a two-sample t-test for a difference in means or a paired t-test for a mean difference? Briefly explain your choice.
A transportation researcher wants to compare the mean daily commute times of office workers in Chicago ($n_1 = 18$) and Seattle ($n_2 = 25$). Assuming the population…
A transportation researcher wants to compare the mean daily commute times of office workers in Chicago ($n_1 = 18$) and Seattle ($n_2 = 25$). Assuming the population distributions are roughly normal but the population variances are not assumed to be equal, what is the conservative number of degrees of freedom the researcher should use for a two-sample t-test?
A consumer advocacy group wants to compare the continuous video playback battery life of two flagship smartphones, Model X and Model Y. They test a random sample of…
A consumer advocacy group wants to compare the continuous video playback battery life of two flagship smartphones, Model X and Model Y. They test a random sample of phones of each model under identical brightness and network conditions.
-
Model X: $n = 35$, $\bar{x} = 14.2$ hours, $s = 1.6$ hours.
-
Model Y: $n = 40$, $\bar{x} = 13.1$ hours, $s = 1.4$ hours.
State the hypotheses to test if there is a significant difference in mean battery life between the two models. Calculate the test statistic. Assuming a conservative degrees of freedom of $34$, the resulting two-tailed P-value is between $0.002$ and $0.005$. State your formal conclusion at the $\alpha = 0.01$ significance level in the context of the problem.
Errors and power
A test can reach the wrong conclusion even when done perfectly, because it is based on a random sample.
| $H_0$ is true | $H_a$ is true | |
|---|---|---|
| Reject $H_0$ | Type I error | Correct decision (power) |
| Fail to reject $H_0$ | Correct decision | Type II error |
In a context question, describe the error in words about the situation. For example, if $H_0$: "the drug does not lower blood pressure" and $H_a$: "the drug lowers it," a Type I error is concluding the drug works when it really does not, and a Type II error is concluding there is no evidence it works when it actually does.
The critical value splits the horizontal axis into "reject" (right) and "fail to reject" (left). Under $H_0$, the area to the right is $\alpha$. Under a specific true alternative, the area to the left is $\beta$ and the area to the right is the power.
Choosing $\alpha$ is a tradeoff based on consequences. If a Type I error is very costly or dangerous, use a small $\alpha$. If a Type II error is the greater danger (missing a real hazard or a real cure), accept a larger $\alpha$ in exchange for more power.
A medical researcher is testing a newly developed drug designed to reduce blood pressure in patients with hypertension. $H_0$: The drug does not reduce blood pressure.…
A medical researcher is testing a newly developed drug designed to reduce blood pressure in patients with hypertension.
-
$H_0$: The drug does not reduce blood pressure.
-
$H_a$: The drug effectively reduces blood pressure.
Describe exactly what a Type I error would represent in the context of this medical study.
A manufacturing plant produces heavy-duty steel cables used in elevator construction. The safe breaking strength of these cables is claimed to be at least $15,000$ lbs.…
A manufacturing plant produces heavy-duty steel cables used in elevator construction. The safe breaking strength of these cables is claimed to be at least $15,000$ lbs. A safety inspector tests a random sample of cables using the following hypotheses:
-
$H_0: \mu = 15,000$ lbs
-
$H_a: \mu < 15,000$ lbs
Describe the practical consequences of making a Type I error and a Type II error in this context. Which error poses a greater risk to public safety?
A local school board funded an expensive pilot program intended to improve standardized test scores. After one year, a two-sample t-test comparing students in the…
A local school board funded an expensive pilot program intended to improve standardized test scores. After one year, a two-sample t-test comparing students in the program to those who were not yielded a P-value of $0.22$. During a public meeting, the board president stated, "This high P-value proves that the pilot program is completely ineffective and has zero impact on test scores."
Statistical software indicated that the power of this specific test to detect a meaningful improvement of $5$ points was only $0.35$. Explain why the board president's conclusion is statistically flawed, referencing both the correct interpretation of a non-significant P-value and the implications of the test's low power.
Summary
Back to Jayden: assuming he really shoots $70\%$, getting $28$ or fewer makes out of $50$ has a probability of only about $0.015$. That is below $\alpha = 0.05$, so we have convincing evidence that his true rate is below $70\%$, and not just an off night. The $1.5\%$ chance is still not zero, so this decision could be a Type I error.
| Procedure | Parameter | Test statistic | Conditions |
|---|---|---|---|
| One-proportion $z$ | $p$ | $z = \dfrac{\hat{p} - p_0}{\sqrt{p_0(1 - p_0)/n}}$ | Random; $10\%$; $np_0, n(1 - p_0) \ge 10$ |
| Two-proportion $z$ | $p_1 - p_2$ | $z = \dfrac{\hat{p}_1 - \hat{p}_2}{\sqrt{\hat{p}_c(1 - \hat{p}_c)(1/n_1 + 1/n_2)}}$ | Random; $10\%$; large counts with $\hat{p}_c$ |
| One-sample $t$ | $\mu$ | $t = \dfrac{\bar{x} - \mu_0}{s/\sqrt{n}}$, $df = n - 1$ | Random; $10\%$; normal or $n \ge 30$ |
| Paired $t$ | $\mu_d$ | $t = \dfrac{\bar{x}_d - 0}{s_d/\sqrt{n}}$, $df = n - 1$ | Random; $10\%$; differences normal or $n \ge 30$ |
| Two-sample $t$ | $\mu_1 - \mu_2$ | $t = \dfrac{\bar{x}_1 - \bar{x}_2}{\sqrt{s_1^2/n_1 + s_2^2/n_2}}$ | Independent random samples; $10\%$; normal or $n \ge 30$ each |