Why does how we collect data decide whether the data can tell us the truth?
Topic 3.1 Introducing Statistics: Do the Data We Collected Tell the Truth? Recognize that the method of data collection determines the kinds of conclusions that can be drawn, and that poorly collected data cannot be fixed by analysis.
A focused answer to AP Statistics Topic 3.1, on why the data-collection method determines what conclusions are valid, the difference between random error and bias, and why analysis cannot rescue badly collected data.
Reviewed by: AI editorial process; not yet individually human-reviewed
Have a quick question? Jump to the Q&A page
Jump to a section
What this topic is asking
The College Board (Topic 3.1) wants you to understand that how data are collected determines what conclusions are valid. Good analysis cannot rescue data gathered by a flawed method, so before any calculation you must ask whether the data can tell the truth at all.
Why the method, not the analysis, comes first
The crucial difference is what fixes each. Random error is reduced by taking a larger random sample, because chance variation averages out. Bias is not reduced by size, because the systematic flaw repeats in every additional observation. A poll that only reaches landline phones excludes mobile-only households no matter how many landlines it dials. This is why the College Board insists that you evaluate the collection method before trusting any statistic computed from the data.
The two questions the method decides
These two questions, generalize? and cause? recur throughout Unit 3, and Topic 3.7 ties them together formally. The point of 3.1 is to instil the habit of asking them at the start, because the answers were fixed the moment the data were collected.
How data go wrong
A study can mislead in many ways, but the common thread is a systematic mismatch between who (or what) was measured and who (or what) the conclusion is about. A voluntary response sample (call-ins, online polls) over-represents people with strong opinions. A convenience sample (whoever is easiest to reach) over-represents the accessible. Nonresponse skews results toward those willing to answer. Undercoverage leaves part of the population with no chance of selection. In every case the flaw is in the method, so the resulting numbers, however carefully analyzed, describe a distorted picture. Recognizing that "garbage in, garbage out" is a statistical principle, not just a slogan, is the heart of Topic 3.1: the most important quality check on a dataset happens before any mean or proportion is computed, by interrogating how the data came to exist.
Why good data are worth the effort
The flip side is that data collected well are extraordinarily powerful. A modest random sample of a few hundred, properly selected, can estimate a national proportion to within a few percentage points, because random selection makes the sample a fair miniature of the population and random error is quantifiable and shrinkable. A randomised experiment, even a small one, can establish causation that no amount of observational data can. So Topic 3.1 is not only a warning about bad data; it is the motivation for the careful sampling (Topics 3.2 to 3.4) and experimental design (Topics 3.5 to 3.7) that make the rest of the unit, and all of the inference units that follow, trustworthy.
Try this
Q1. Explain the difference between bias and random error, and which one a larger sample reduces. [2 points]
- Cue. Bias is a systematic error in one direction from a flawed method; random error is chance sample-to-sample variation. A larger random sample reduces random error but not bias.
Q2. Why can a flawed sampling method not be fixed by collecting more data? [1 point]
- Cue. The systematic flaw repeats in every observation, so a bigger sample is just as biased; only changing the method removes the bias.
Exam-style practice questions
Practice questions written in the style of College Board exam questions on this dot point, with worked answer explainers. The year tag is the paper they imitate, not the source.
AP 2019 (style)1 marksSection I (multiple choice). A radio host asks listeners to call in and vote on a proposal. Which is the most serious problem with using the result to describe public opinion? (A) The sample size is too small (B) The arithmetic of the percentages is wrong (C) The sample is self-selected and unrepresentative (D) The proposal is too complicatedShow worked answer →
The correct answer is (C).
A call-in (voluntary response) sample is made up of people who choose to participate, who tend to hold stronger or more extreme opinions, so it is systematically unrepresentative of the public. This is bias built into the collection method.
(A) A large self-selected sample is still biased; size does not fix bias. (B) The issue is the data, not the arithmetic. (D) Complexity is not the core flaw. The collection method, not analysis, determines whether the data can tell the truth.
AP 2021 (style)4 marksSection II (free response). A company emails a satisfaction survey to all customers; reply, and of those, report being satisfied. (a) Explain why the may not reflect the satisfaction of all customers. (b) Identify the type of bias most likely present, and justify your answer in context. (c) Explain why collecting more responses in the same way would not necessarily fix the problem.Show worked answer →
A 4-point question on collection method and bias.
(a) (1 point) Only replied, and those who reply may differ systematically from those who do not (for example, the very satisfied or very unsatisfied are more motivated to respond), so the describes responders, not all customers.
(b) (2 points) Nonresponse bias (1 point): the large majority who did not reply may hold different views, and because responding is voluntary the respondents are self-selected, so the result is skewed toward whoever chose to answer (1 point, in context).
(c) (1 point) More responses gathered the same way keep the same self-selection, so the bias persists; a representative result needs a different method (such as following up a random sample of non-responders), not just a bigger biased sample.
Markers reward recognizing that responders may differ from non-responders, naming nonresponse (voluntary response) bias in context, and the insight that size does not cure bias.
Related dot points
- Topic 3.2 Introduction to Planning a Study: distinguish observational studies from experiments, identify explanatory and response variables, and recognize that only an experiment with imposed treatments can support a causal conclusion.
A focused answer to AP Statistics Topic 3.2, distinguishing observational studies from experiments, identifying explanatory and response variables and confounding, and explaining why imposing treatments is what enables causal claims.
- Topic 3.3 Random Sampling and Data Collection: describe and distinguish simple random, stratified, cluster, and systematic random sampling, and explain why random selection supports generalization to a population.
A focused answer to AP Statistics Topic 3.3, describing simple random, stratified, cluster, and systematic random sampling, how each uses chance, their trade-offs, and why random selection allows generalization, with a worked SRS selection.
- Topic 3.4 Potential Problems with Sampling: identify undercoverage, voluntary response, convenience, nonresponse, and response bias, explain how each distorts results, and recognize that bias is not reduced by a larger sample.
A focused answer to AP Statistics Topic 3.4, identifying undercoverage, voluntary response, convenience, nonresponse, and response bias, the direction each pushes results, and why bias persists no matter how large the sample.
- Topic 3.7 Inference and Experiments: use the presence or absence of random selection and random assignment to determine the scope of inference, that is, whether results generalize to a population and whether a causal conclusion is justified.
A focused answer to AP Statistics Topic 3.7, on the scope of inference, using random selection (generalization) and random assignment (causation) to decide what conclusions are valid, with a worked four-quadrant analysis.
- Topic 1.1 Introducing Statistics - What Can We Learn from Data?: identify questions to be answered, based on variation in one-variable data, and recognize what a data set can and cannot tell us.
A focused answer to AP Statistics Topic 1.1, on how variation in data raises statistical questions, what kinds of question data can answer, and the limits of what a single data set reveals, with worked examples.
Sources & how we know this
- AP Statistics Course and Exam Description — College Board (2020)