Exercises: Sampling and experimental design

A high school principal wants to survey students about a proposed change to the school's dress code.

• Scenario A: The principal randomly selects two homerooms and surveys every student in those homerooms.

• Scenario B: The principal divides the student body by grade level (freshmen, sophomores, juniors, seniors) and randomly selects $25$ students from each grade to survey.

Identify the sampling methods used in Scenario A and Scenario B. Briefly explain the primary difference in how the groups are utilized in these two approaches.

A local city council mails a survey to $5,000$ randomly selected residents to ask for their opinions on funding a new $2.5$ million dollar community pool. The survey includes a pre-paid return envelope. After one month, only $450$ surveys are returned. The results show that $85\%$ of the respondents strongly support building the new pool.

Identify the primary source of bias in this survey. Explain how this bias might affect the council's conclusion about the true level of support in the community.

A researcher is interested in whether drinking green tea reduces blood pressure. She recruits $100$ volunteers and asks them to record how many cups of green tea they drink per week. At the end of the month, she measures their blood pressure and looks for a relationship between tea consumption and blood pressure levels.

Is this an observational study or an experiment? Justify your answer based on the actions taken by the researcher.

An agricultural scientist is testing a newly developed pesticide on corn crops. She takes $20$ plots of land of equal size and randomly assigns $10$ plots to be sprayed with the new pesticide and $10$ plots to be sprayed with a standard, widely used pesticide. After one month, she counts the number of surviving pests in each plot.

Identify the experimental units, the treatments, and the response variable in this study.

A tech company in Silicon Valley has $800$ employees. The human resources manager wants to select a sample of $40$ employees to ask about workplace satisfaction. On a Monday morning, he stands by the main entrance and surveys the first $40$ employees who walk into the building.

Explain why this method does not produce a Simple Random Sample (SRS). Describe a valid procedure the manager could use to obtain a true SRS of $40$ employees.

A pharmaceutical company is testing a new allergy medication. They recruit $200$ allergy sufferers and randomly assign $100$ to receive the new medication and $100$ to receive a placebo pill that looks and tastes exactly like the real medication. The patients do not know which pill they are taking, but the doctors administering the pills and recording the symptoms do know.

Is this a single-blind or double-blind experiment? Explain the specific purpose of using a placebo in this context.

A political action committee conducts a phone survey regarding a proposed tax increase. They ask the following question: "Given the recent alarming rise in violent crime in our city, do you support increasing funding for the police department through a small property tax adjustment?"

Identify the type of bias present in this survey question. Explain how the phrasing could artificially inflate the proportion of people who answer "Yes."

A quality control inspector at a factory needs to check the weight of cereal boxes coming off an assembly line. She randomly selects a number between $1$ and $15$, which turns out to be $7$. She then inspects the $7$th box and every $15$th box after that (the $22$nd, $37$th, $52$nd, etc.).

Name this sampling method. State one practical advantage this method has over a simple random sample in a fast-paced industrial setting.

A sports apparel company has developed a new running shoe material designed to reduce wear and tear. They want to compare it to their standard material. They recruit $30$ marathon runners to participate in a study.

Design a matched pairs experiment to test which material is more durable. Explain exactly how randomization is incorporated into your design and why a matched pairs design is superior to a completely randomized design (where $15$ runners get the new shoes and $15$ get the standard shoes) for this specific study.

A researcher wants to study the effect of a new digital teaching platform on standardized test scores. She knows that a student's prior math ability (high, medium, or low) strongly influences their test scores regardless of the teaching method.

Explain the conceptual difference between blocking and stratifying in statistical design. If she is designing an experiment to test the teaching platform, which technique should she use to account for prior math ability, and how exactly should she implement it?

A popular health magazine publishes an article claiming that "Eating oatmeal for breakfast causes people to live longer." This bold conclusion was based on a large-scale observational study showing a strong positive association between the frequency of eating oatmeal and life expectancy.

Explain why the magazine's claim of causation is statistically flawed. Identify a specific potential confounding variable (also known as a lurking variable) and explain how it could account for the observed association between oatmeal consumption and longer life.

In a clinical trial for a new weight-loss drug, $150$ volunteers are randomly assigned to either the treatment group (receiving the drug) or the control group (receiving a placebo).

Explain the specific purpose of random assignment in this experiment. What potential problem does random assignment help to prevent, and how does it allow researchers to draw conclusions about cause and effect?

A local radio station wants to estimate the proportion of residents in the county who support a proposed property tax increase to fund public schools. The station's morning host asks listeners to call a toll-free number and vote "Yes" or "No."

Identify two distinct sources of bias in this sampling method (e.g., undercoverage, voluntary response bias). Explain how each source of bias would likely skew the results regarding the true level of support for the tax increase.

A large city consists of $150$ distinct neighborhoods. The mayor wants to survey residents about a new trash collection schedule. Because it is too expensive to send surveyors to every neighborhood, the mayor's team randomly selects $8$ neighborhoods and surveys every single household within those $8$ chosen areas.

Identify the sampling method used. What is the main logistical advantage of this method in this scenario? Furthermore, describe a potential disadvantage of this method if the city's neighborhoods are highly segregated by income or demographics.

A university professor randomly selects $50$ students from her introductory psychology class to participate in a study on memory. She then randomly assigns $25$ students to study a list of words with background music playing, and $25$ to study in silence. She finds that the group studying in silence scored significantly higher on a subsequent memory test.

Based on the design of this study, can the professor conclude that studying in silence causes better memory performance? Can she generalize these results to all university students? Justify both of your answers by referring to the principles of experimental design.

An ecologist wants to estimate the number of diseased trees in a large, rectangular commercial forest. The trees are planted in perfectly straight rows of $50$ trees each. The ecologist decides to use a systematic sample, inspecting every $50$th tree starting from the $10$th tree in the first row.

Explain why this specific systematic sampling plan might yield a highly biased estimate of the overall disease rate if the disease tends to spread easily along the outer edges of the forest where trees are exposed to more wind.

An agronomist is testing four different fertilizer blends (A, B, C, and D) on the yield of a specific variety of wheat. The testing field is located on a hillside, meaning the plots at the top of the hill receive less water runoff and are drier than the plots at the bottom. There are $12$ plots available in total: $4$ at the top, $4$ in the middle, and $4$ at the bottom.

Design a randomized block experiment to test the fertilizers. Explain why blocking by hill position is necessary in this context, and detail how you would assign the treatments to the plots.

A federal education agency wants to estimate the average amount of student loan debt for graduating college seniors across the entire United States. Creating a single master list of all graduating seniors in the country to draw an SRS is impossible.

Describe a multistage sampling design that the agency could use to select a representative sample of $2,000$ graduating seniors. Include at least three distinct stages of sampling, and specify the sampling method (e.g., simple random, cluster, stratified) used at each stage.

A tech company wants to test a new ergonomic keyboard against their standard keyboard to see if it increases typing speed. They recruit $60$ employees. The first $30$ employees who arrive at work on Monday morning are given the ergonomic keyboard. The next $30$ employees to arrive are given the standard keyboard. They track the typing speed of both groups for a week. The group with the ergonomic keyboard showed a $15\%$ increase in typing speed.

Identify the major flaw in this experimental design. Explain how a specific confounding variable might be present due to this flaw, and describe how you would redesign the experiment to eliminate it.

A dermatologist is testing a new topical ointment designed to clear up mild eczema. She has $40$ patients available for the study, all of whom have mild eczema patches on both of their arms.

Compare a completely randomized design to a matched pairs design for this study. Describe exactly how each design would be implemented. Which design provides more statistical power to detect a true difference between the new ointment and a placebo, and why?

A major news network conducts an online poll on its homepage asking, "Do you believe the current administration is handling the economy well?" Over $100,000$ people respond, with $72\%$ saying "No." Meanwhile, a reputable polling organization conducts a simple random sample of $1,000$ registered voters via phone calls, but only $120$ people answer the phone and complete the survey. Of those $120$ respondents, $51\%$ say "No."

Discuss the specific types of bias present in both polls. Which result, if either, is more likely to accurately reflect the true proportion of the population? Justify your reasoning by comparing the impact of voluntary response bias versus nonresponse bias.

Consider the "Scope of Inference" matrix, which dictates what conclusions can be drawn based on study design. Evaluate the following four scenarios:

1. Random selection of participants, random assignment to treatments.

2. Random selection of participants, no random assignment.

3. No random selection (e.g., volunteers), random assignment to treatments.

4. No random selection, no random assignment.

For each scenario, determine whether it is statistically valid to make an inference about the population, and whether it is valid to establish a cause-and-effect relationship. Provide a brief, real-world example of a study that fits Scenario 3.

A chemical engineer is trying to maximize the yield of a chemical reaction. She suspects that both the temperature ($70^\circ\text{F}$ or $90^\circ\text{F}$) and the stirring rate (Slow, Medium, or Fast) affect the yield. She has $24$ identical reaction vats available for an experiment.

Design a completely randomized experiment with two factors to test her hypothesis. List all the specific treatments (combinations of levels) and explain exactly how you would randomly assign the $24$ vats to these treatments to ensure a balanced design.

A university research hospital is conducting a clinical trial for a new, minimally invasive surgical procedure designed to relieve severe knee pain.

Explain what a "sham surgery" (a surgical placebo) would entail in the context of this experiment. Discuss the scientific benefits of using a sham surgery to establish a control group and maintain a double-blind environment. Finally, contrast these scientific benefits with the ethical concerns of subjecting patients in the control group to the risks of anesthesia and incisions without providing the actual therapeutic treatment.