Exercises: Survey methodology sheets and critical reading

An election poll publishes the following methodology sheet: "Target population: electoral roll of the province ($650.000$ people). Sample: $1.200$ interviews. Margin of error: $\pm 2,8\ \%$. Confidence level: $95,5\ \%$".

• a) Explain what each of the four items of data means.

• b) What is the relationship between the confidence level and the certainty that the true result lies within the margin of error?

True or false, justifying your answer: "If a survey was conducted with $1.200$ people and a confidence level of $95\ \%$, its result coincides exactly with the true value for the entire population."

A survey on reading habits gives $34\ \%$ daily readers, with a margin of error of $\pm 3\ \%$ and a confidence level of $95 \%$.

• a) Between which values does the true percentage of daily readers lie, at that confidence level?

• b) A headline claims: "Exactly $34$ out of every $100$ people read daily". Correct the headline so that it is honest.

To find out about library usage in a secondary school with $900$ students, $50$ students from a single year group—those in the study hall during the first period—are surveyed.

• a) Identify which element of the methodology sheet is flawed: the target population, the sample, or the calculation of error?

• b) Explain why the sample is not representative and propose how to select it correctly.

Lucía is preparing a social research project and needs to conduct two surveys with a $95\,\%$ confidence level ($z_{\alpha/2} = 1,96$) and a maximum margin of error of $\pm 3\,\%$ in the worst-case scenario ($\hat{p} = \hat{q} = 0,5$). One of the surveys will target the population of Madrid (approximately $3.300.000$ inhabitants) and the other that of Valladolid (around $300.000$ inhabitants). Lucía claims: “Since the population of Madrid is about $11$ times larger than that of Valladolid, I will need to interview $11$ times more people in Madrid to achieve the same margin of error.”\n- a) Find the flaw in Lucía's reasoning and explain why she is mistaken based on the formula for the sampling error $E = z_{\alpha/2}\sqrt{\frac{\hat{p}\hat{q}}{n}}$.\n- b) Calculate the required sample size $n$ for each of the two cities under the required conditions (round up to the next integer). Does the required sample size vary between both cities?\n- c) In what sole situation would the size of the universe appreciably influence the required sample size?

A polling company publishes an opinion survey conducted on a random sample of $n = 2.500$ adults in Spain.\n- a) Before doing detailed calculations, mentally estimate the sampling margin of error $E$ (in percentage) corresponding to this sample for a $95\,\%$ confidence level, applying the quick worst-case rule $E \approx \frac{1}{\sqrt{n}}$.\n- b) Calculate the exact margin of error using the formula $E = z_{\alpha/2}\sqrt{\frac{0,25}{n}}$ with $z_{\alpha/2} = 1,96$. Compare the result with your previous estimate and calculate the difference between both values.\n- c) In light of the results, assess whether the quick rule $1/\sqrt{n}$ is a reasonable and useful tool for an ordinary citizen to critically interpret the technical specifications appearing in the press.

An online newspaper runs the headline: "$60\ \%$ of young people reject the council's new app". In the small print, it reads: survey of $80$ users of a municipal services complaint forum.

• a) Point out all the methodological issues with this headline (sample selection, size, and bias).

• b) Using the margin of error formula $E = 1,96 \cdot \sqrt{\frac{0,5 \cdot 0,5}{n}}$, estimate the margin of error that even a random sample of that size would have.

• c) Rewrite the headline honestly.

The methodology sheet of an opinion poll states: "Target population: $2.400.000$ voters. Sample: $2.400$ random interviews. Margin of error: $\pm 2\ \%$. Confidence level: $95\ \%$".

• a) Interpret each piece of data in the methodology sheet.

• b) Explain qualitatively why the margin of error depends on the sample size and hardly at all on the population size.

• c) The poll gives $21\ \%$ to one party and $22\ \%$ to another. Can it be claimed that the second party is ahead? Justify your answer using the margin of error.

In a report on urban mobility in a Spanish capital, the following headline appears: "Support for expanding bike lanes stands between $55,5\,\%$ and $60,5\,\%$ of citizens". The technical sheet accompanying the news item only states: "Confidence level: $95\,\%$ ($z_{\alpha/2} = 1,96$). Simple random sampling. Worst-case scenario ($\hat{p} = \hat{q} = 0,5$)".\n- a) From the endpoints of the confidence interval provided in the news item, reconstruct the estimated sample proportion $\hat{p}$ and the margin of error $E$ used by the sociological team.\n- b) Deduce the exact sample size $n$ that was interviewed to produce the study (round up to the next integer).\n- c) A councilor states in a press conference: "We can state with total mathematical certainty that more than half of the citizens support the measure". Is their statement rigorous from the point of view of statistical inference? Justify your answer by distinguishing between absolute certainty and confidence level.

The Municipal Youth Institute of a city with $50.000$ registered young people (aged 18 to 30) wishes to estimate the youth emancipation rate using a survey. They have a maximum grant of $7.200$ euros, and the operational cost of conducting and recording each in-person questionnaire is $8$ euros.\n- a) If the technicians require a confidence level of $95,5\,\%$ ($z_{\alpha/2} = 2$) and a maximum margin of error of $\pm 2,5\,\%$ in the worst-case scenario ($\hat{p} = \hat{q} = 0,5$), calculate the minimum number of interviews needed and determine, with justification, whether the allocated budget is sufficient.\n- b) Given the maximum available budget ($7.200$ euros), calculate the minimum margin of error they can afford while maintaining the $95,5\,\%$ confidence level.\n- c) To collect the data, a technician proposes positioning interviewers at the gates of the city's public university on a Tuesday at $11:00$. Model an alternative sampling proposal that avoids the biases of this procedure and justify which demographic variables it would be advisable to stratify by.

Two surveys on the same population and in the same week give different results for the same event: the first estimates $38\ \%$ with a margin of error of $\pm 2\ \%$ (sample of $2.500$ people) and the second, $34\ \%$ with a margin of error of $\pm 4,5\ \%$ (sample of $450$ people), both at a $95\ \%$ confidence level.

• a) Write down the intervals in which the true value would lie according to each survey and check that they overlap.

• b) Using the formula for margin of error, explain why the second has a much larger margin.

• c) If a headline reads "The surveys contradict each other", is it correct? Justify your answer.

You want to estimate the percentage of students in your city ($25.000$ students) who eat a proper breakfast, with a maximum margin of error of $\pm 3\ \%$ and a confidence level of $95 \%$.

• a) Calculate the sample size required in the worst-case scenario ($p = q = 0,5$), using $n = \frac{1,96^2 \cdot 0,25}{E^2}$.

• b) Describe the population (universe) and a sampling procedure that guarantees representativeness.

• c) Write out the full methodology sheet (factsheet) of your study as it would appear published.

During a debate on store opening hours in an autonomous community, two media outlets publish their respective polls on the same day:\n- Method A (Open web poll): A radio station invites people to vote on its website for $24$ hours. $40.000$ users participate, and $78\,\%$ oppose opening on Sundays. The media outlet publishes on its front page: "The error for $40.000$ responses is barely $\pm 0,5\,\%$ at a $95\,\%$ confidence level".\n- Method B (Probability sampling): A polling institute conducts telephone interviews with $1.200$ people selected via stratified random sampling by age and municipality. The result is $54\,\%$ against, with a margin of error of $\pm 2,8\,\%$ at a $95\,\%$ confidence level ($z_{\alpha/2} = 1,96$, worst-case scenario).\n- a) Check analytically whether the numerical calculation of the sampling error in Method A ($E \approx 1/\sqrt{n}$) is correct for $n = 40.000$ and explain why applying this mathematical formula to such a poll constitutes a methodological fallacy.\n- b) Compare both methods and justify which of the two provides a valid scientific basis for inferring the opinion of the autonomous community's population. In your answer, analyze the concepts of self-selection, sampling frame, and representativeness.\n- c) Formally refute the following statement, which is common in public debates: "A voluntary sample of $40.000$ citizens always has greater statistical rigor than one of only $1.200$ people, because it collects nearly forty times more real opinions".

A sociological consultancy conducts a study on recycling habits in a region based on a representative random sample of $n = 1.000$ people, with a confidence level of $95\,\%$ ($z_{\alpha/2} = 1,96$). The overall technical sheet shows a sampling error of $E = \pm 3,1\,\%$ in the worst-case scenario. The sample consists of $500$ women and $500$ men. When asked whether they separate organic waste, $53\,\%$ of the women and $47\,\%$ of the men state that they do so regularly.\nMarcos, a trainee analyst, writes the following two conclusions in his report:\n1. "Since the survey error is $\pm 3,1\,\%$, the confidence interval for women is $[49,9\,\%; 56,1\,\%]$ and for men is $[43,9\,\%; 50,1\,\%]$".\n2. "Given that both intervals overlap between $49,9\,\%$ and $50,1\,\%$, it is formally proven that there is no significant difference between the behavior of women and men".\n- a) Point out the error made by Marcos in item 1. Calculate the real margin of error for each subgroup ($n = 500$) in the worst-case scenario at a $95\,\%$ confidence level and write the corrected confidence intervals.\n- b) Refute Marcos's reasoning in item 2 by explaining the two conceptual errors made: why the overlap of two individual intervals does not by itself decide whether a difference is statistically significant, and why the absence of statistical significance is not equivalent to proving equality.\n- c) If the consultancy wanted the specific margin of error for the subgroup of men to be at most $\pm 2,5\,\%$ at a $95\,\%$ confidence level (worst-case scenario), how many men would need to be interviewed in total?

Unit Challenge. Does your environment reflect national statistical reality? You will investigate an official statistic about youth and contrast it with your own data.

1. Official fact sheet: Check the INE or CIS website for a recent study on youth habits (e.g., percentage of frequent readers or weekly sports practice, $p_0$). Extract and analyse the elements of its technical specifications: sample size, confidence level, and margin of error allowed.

2. Sampling and interval: Ask $n = 30$ young people in your environment the same dichotomous question. In a spreadsheet or GeoGebra, calculate the sample proportion $\hat{p}$ and find the $95\,\%$ confidence interval for the population proportion ($z_{\alpha/2} = 1,96$).

3. Hypothesis testing and critical reading: Set up a two-tailed hypothesis test ($H_0: p = p_0$ versus $H_1: p \neq p_0$) with a significance level $\alpha = 0,05$. Make a well-founded statistical decision. Finally, prepare a one-page executive report discussing whether your environment differs from the Spanish average, assessing the biases of your sample against the rigour of the official survey.

Evaluation criteria:

• Identification and critical analysis of the elements of the official fact sheet.

• Accuracy in calculating the confidence interval and formal execution of the hypothesis test using software.

• Quality of the critical reading regarding selection bias, representativeness, and conclusions.