Week 05 · Marketing Research & Analytics
Davood Wadi, PhD
The conclusions of a survey extend only as far as the population that its sample represents.
The population is the complete set of elements about which the researcher seeks information, such as all households in a city or all customers of a firm.
A census collects data from every element of the population.
A sample is a subset of the population, selected to draw conclusions about the population as a whole.
A census is appropriate when the population is small, as in many business-to-business markets, or when the variation among elements is large.
Samples are used rather than censuses when the population is large, when time and budget are limited, and when a census would not substantially improve the accuracy of the findings.
The researcher first defines the target population, specifying the elements, the geographical boundaries, and the time period.
The researcher then determines the sampling frame, which is the list or procedure from which the sample is drawn, such as a customer database or a panel of registered respondents.
The researcher then selects a sampling technique.
The researcher then determines the sample size.
Finally, the researcher executes the sampling process and documents how it was carried out.
For example, the target population of a study for a regional bank may be defined as account holders aged 18 or older who used the bank's mobile application in the past three months.
A supermarket chain plans a survey of its loyalty-card members. Put the steps of its sampling design in order.
Tap a step, then the number where it belongs.
Sampling error is the difference between a sample result and the true population value that arises because only a subset of the population is observed.
Sampling error decreases as the sample size increases, and can be estimated for probability samples.
Nonsampling error arises from sources other than sampling, and does not decrease as the sample size increases.
Frame error occurs when the sampling frame does not correspond to the target population, as when a list of email subscribers is used to represent all customers.
Nonresponse error occurs when the people who respond differ systematically from those who do not.
Response error occurs when respondents give inaccurate answers, whether because of poorly worded questions, faulty recall, or the response biases introduced in Week 4.
Each case describes a gap between a survey result and the true population value. Name its source.
The list from which the sample is drawn does not correspond to the target population of all residents.
Each sample observes only a subset of the population, so their results differ from each other and from the true value. A larger sample would reduce this gap.
The respondents give inaccurate answers because of faulty recall.
The people who responded differ systematically from those who did not.
Increasing the sample size does not reduce nonsampling error: the list of app users still does not correspond to all customers.
Which concept does this case show?
Sampling techniques are classified as probability or nonprobability techniques.
In probability sampling, every element of the population has a known, nonzero chance of selection.
In nonprobability sampling, selection depends on the judgment of the researcher or the convenience of access, and the chance of selection is unknown.
Only probability samples permit the statistical estimation of sampling error.
In simple random sampling, every element of the population has an equal chance of selection, and every possible sample of a given size is equally likely.
Simple random sampling requires a complete sampling frame, from which elements are selected by a random procedure.
In systematic sampling, the researcher selects a random starting point in the sampling frame and then selects every kth element, such as every 20th customer on a list.
Systematic sampling is simpler to execute than simple random sampling, but it produces a biased sample if the list follows a pattern that coincides with the sampling interval.
In stratified sampling, the population is divided into subgroups, called strata, and a random sample is drawn from each stratum.
Strata are defined so that elements within a stratum are similar, such as customers grouped by region or by spending level.
Stratified sampling increases precision and ensures that small but important subgroups are represented.
In cluster sampling, the population is divided into clusters, such as city blocks or stores, a random sample of clusters is selected, and elements are studied within the selected clusters.
Cluster sampling reduces the cost of data collection, particularly for in-person interviews, but it is less precise than stratified sampling of the same size.
In convenience sampling, elements are selected because they are easily accessible, such as shoppers intercepted in a mall.
In judgmental sampling, the researcher selects elements believed to be representative or especially informative, such as expert users of a product.
In quota sampling, the researcher sets quotas for subgroups, such as age and gender, in proportion to the population, and fills each quota by convenience or judgment.
In snowball sampling, initial respondents identify further respondents, which is useful for rare or hard-to-reach populations.
Nonprobability samples are faster and less expensive than probability samples, and are appropriate for exploratory research and pretesting.
Their results cannot be projected to the population with a known level of precision.
Probability sampling is appropriate when the findings must be projected to the population with a known level of precision, as in estimates of market size.
Nonprobability sampling is appropriate for exploratory research, for pretesting, and when no sampling frame is available.
Stratified sampling is appropriate when subgroups differ substantially and each must be estimated accurately.
Cluster sampling is appropriate when the population is geographically dispersed and in-person data collection is required.
Snowball sampling is appropriate when the population is rare and its members know one another.
Match each research situation with the sampling technique appropriate to it.
Situation
Technique
Tap a term, then the example it goes with.
The sample size is determined by statistical and practical considerations.
Larger samples are required for important decisions, for descriptive and causal research, for analyses of many variables, and for separate estimates of subgroups.
The sample size is also constrained by the available budget and time, and by the proportion of contacted people who qualify for and complete the study.
The precision of an estimate is expressed as a margin of error, such as plus or minus five percentage points.
The confidence level is the probability that the interval formed by the estimate and its margin of error contains the true population value, conventionally 95 percent.
The sample size required to estimate a proportion is n = z² × p(1 − p) / e², where z is 1.96 for 95 percent confidence, p is the expected proportion, and e is the margin of error.
With p set to 0.5, the most conservative assumption, a margin of error of five percentage points at 95 percent confidence requires a sample of 385.
Halving the margin of error requires approximately four times the sample size.
For large populations, the required sample size depends on the desired precision, not on the size of the population.
A survey of 385 customers estimates a proportion with a margin of error of plus or minus five percentage points at 95 percent confidence. The marketing director asks for a margin of plus or minus 2.5 points, at the same confidence level. Approximately how many completed questionnaires are required?
The incidence rate is the proportion of contacted people who qualify for the study.
The completion rate is the proportion of qualified people who complete it.
The number of people who must be contacted equals the required sample size divided by the product of the incidence rate and the completion rate.
For example, a study that requires 385 completed questionnaires, with an incidence rate of 50 percent and a completion rate of 40 percent, requires approximately 1,925 contacts.
Once the sample is designed, the data are collected through a survey method.
The method affects the cost and speed of the study, the population that can be reached, and the accuracy of the responses.
Online surveys are the least expensive and fastest method, but they reach only people with internet access who are willing to participate, and they provide no control over who completes the questionnaire.
Telephone surveys allow an interviewer to clarify questions, but response rates have declined substantially.
In-person surveys allow complex questions, product demonstrations, and the highest level of control, at the highest cost.
Self-administered surveys, whether online or by mail, reduce interviewer bias and social desirability bias on sensitive topics.
Most online surveys in marketing research draw respondents from online panels: large pools of people who have agreed to take part in surveys in exchange for incentives.
Most online panels are opt-in panels, recruited through advertising and referrals. They are nonprobability samples, even when quotas are applied.
Some members of opt-in panels complete a large number of surveys, which may make their responses less representative of ordinary consumers.
Probability-based panels recruit members through random sampling, such as the random selection of residential addresses, and are more costly to maintain.
Speeders complete the questionnaire too quickly to have read the questions.
Straight-liners select the same response option for every item in a set of rating questions.
Inconsistent respondents give answers that contradict one another, such as reporting no purchases in a category and then rating a brand in that category.
Fraudulent respondents misrepresent their characteristics to qualify for surveys, or complete the same survey several times.
Bots are automated programs that complete surveys to collect incentives. Bots that use large language models can produce fluent answers to open-ended questions.
Completion time is compared with a minimum threshold, and respondents who complete the questionnaire implausibly quickly are removed.
Attention checks instruct respondents to select a specific answer, such as “select ‘disagree’ for this item,” and respondents who fail them are removed.
Consistency checks compare answers to related questions and flag contradictions.
Open-ended answers are reviewed for irrelevant, copied, or generic text, including text that appears to be generated by AI.
Technical checks identify duplicate responses from the same device and automated activity.
The criteria for removing respondents are specified before data collection, and the number of respondents removed is reported.
Each case describes a respondent in an online survey. Name the check that would identify them.
Completion far below the minimum threshold shows that the questions could not have been read.
The answers to related questions contradict one another.
The respondent failed an instruction to select a specific answer.
Technical checks identify duplicate responses from the same device.
Generic text that could answer any question, possibly generated by AI, is found by reviewing the open-ended answers.
Which concept does this case show?
Discussion: A meal-kit company plans to estimate the proportion of households in a city that would subscribe to a new service. The marketing team proposes a survey of the company's social media followers. Which errors could this sample introduce, and which alternative sampling design would support a more accurate estimate?
Synthetic respondents are AI-generated answers to survey questions, produced by prompting a large language model to respond as a person with specified characteristics.
A set of synthetic respondents is sometimes called a silicon sample.
Commercial vendors now offer synthetic respondents as a faster and less expensive alternative to human samples.
The researcher specifies a profile for each synthetic respondent, such as age, gender, income, and region.
The language model is instructed to answer the questionnaire as a person with that profile.
Many profiles are generated to match the composition of the target population, and the answers are aggregated.
Studies have found that synthetic respondents can reproduce some aggregate patterns of real survey responses, such as differences in opinion between broad demographic groups.
A synthetic sample is not drawn from the target population. Its responses reflect the data on which the model was trained, and no sampling frame connects them to the population.
Synthetic responses typically show less variation than real responses, which understates the diversity of opinion within the population.
Synthetic responses may reproduce stereotypes about demographic groups rather than the actual views of their members.
Synthetic responses are sensitive to the wording of the prompt, so that minor changes in instructions can change the results.
Synthetic respondents have no experience of products, prices, or events that are absent from the model's training data, which limits their value for new products.
The margin of error and confidence level introduced in this week cannot be meaningfully calculated for a synthetic sample.
A vendor's synthetic panel of 5,000 respondents rates a proposed plant-based cheese at 5.1 on a 7-point purchase-intention scale, with a standard deviation of 0.4, and reports a margin of error of ±0.01. What is the most serious problem with this result?
Synthetic respondents can be used to pretest questionnaires, as introduced in Week 4.
They can be used in exploratory research to generate hypotheses for testing with real respondents.
They are not used as a substitute for real respondents when the findings inform consequential decisions.
Where synthetic respondents are used, their results are compared with those of a real sample on the same questions, and their use is disclosed in the report.
A language model generates its responses with controlled randomness, so that the same prompt can yield different responses.
Each response is therefore a draw from the model's response distribution, in the way that each respondent is a draw from a population.
The common practice of evaluating a model on a single response to each prompt ignores this variation.
In one evaluation, the willingness to pay that language models stated for the same product under the same prompt varied with a coefficient of variation of up to 0.44 (Wadi & Fredette, 2025).
The mean of many responses to the same prompt is an unbiased estimate of the model's typical behavior, and its standard error decreases as the number of responses increases (Wadi & Fredette, 2025).
An analyst asks a language model once how much it would pay for a one-kilogram bag of coffee, and it answers $14. The analyst reports that the model values the product at $14. A colleague submits the same prompt 100 times and obtains a mean of $11.40 with a standard deviation of $4.10. Which statement is correct?
The number of responses per prompt is determined in the same way as a sample size: by the required precision, or by the power to detect an effect of a given size.
For example, a power analysis can determine the number of responses required to detect a small difference among several models or prompts (Wadi & Fredette, 2025).
Prompts are also sampled. Because results can change with minor changes in wording, conclusions are more general when they hold across several wordings of the same instruction.
Models are sampled as well. A result obtained with one model does not generalize to other models, so evaluations include several models from several providers.
Discussion: A vendor offers a synthetic panel of 10,000 respondents, delivered within one hour at a fraction of the cost of a human sample, and states that its results are “95 percent accurate.” Which questions should the research team ask the vendor, and for which decisions, if any, would such a panel be appropriate?
A sample is selected from a sampling frame to draw conclusions about a defined target population.
Sampling error decreases as the sample size increases; nonsampling errors, such as frame, nonresponse, and response errors, do not.
Only probability samples permit the estimation of sampling error, while nonprobability samples are appropriate for exploratory research and pretesting.
The sample size depends on the desired precision and confidence, and the number of contacts depends on the incidence and completion rates.
Online panels require data quality checks for speeders, straight-liners, fraudulent respondents, and bots, with criteria specified in advance.
Synthetic respondents are not a sample of the target population, and they supplement rather than replace real respondents.
A single model response is not a measurement: AI evaluations sample many responses per prompt, determine their number by precision or power, and vary prompts and models.