When are synthetic respondents valid substitutes for human survey data?

Population distributions, concept rankings, digital twins and the limits of simulated consumer evidence

This retrospective review considers source versions available by 7 October 2026. The cutoff defines the evidence included; preparation and source checking took place on 21 October 2026.

Keywords: synthetic respondents; survey validity; digital twins; consumer research

1. What does it mean to replace a respondent?

Synthetic survey research uses a model to produce answers that would otherwise be collected from people. The model may receive a demographic description, a detailed personal interview or an individual’s earlier questionnaire responses. These inputs define different prediction tasks. A model asked to imitate an age group is not being tested on the same information as an agent grounded in a two-hour interview with one person. Their reported accuracy cannot be compared without preserving that difference.

This review asks which kinds of human-survey validity have been established for synthetic respondents. It separates agreement in population averages, rankings of concepts, subgroup differences and individual-level predictions. It also distinguishes reproducing an earlier dataset from estimating a new population quantity. A method can approximate a known average while misrepresenting the distribution that produced it; it can rank familiar product concepts while missing the preferences of a minority segment.

The central issue is the target of inference. If a question concerns what a particular model generates under a prompt, model samples are direct observations of that system. If the question concerns what consumers believe or would choose, generated answers are predictions requiring external validation. Plausible wording cannot bridge that inferential gap. Neither a realistic persona narrative nor a large synthetic panel establishes that the panel represents people who were never observed.

2. Review method

Targeted searches on 21 October 2026 covered synthetic respondents, silicon samples, digital twins, survey validity, cultural alignment and purchase-intent elicitation. The review consulted original publisher records, proceedings abstracts, accessible author manuscripts and Pew Research Center’s September 2026 report. 19 sources were selected to represent contrasting findings and measurement targets. The selection is narrative and purposive; it is not an exhaustive systematic review or a meta-analysis.

Version control is material. Park and colleagues’ third version uses the title LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals. Horton and colleagues’ revised record lists additional authors compared with the initial circulation. The references identify the versions actually consulted. Numerical outcomes reported in earlier summaries are not imported where the current abstract does not establish them [3] [12].

The synthesis records what information the simulation receives and what outcome is held out for evaluation. It avoids a single pooled accuracy score because the studies measure different targets. Full-text access was available for selected author manuscripts and the Bisbee publisher article; several other reports were consulted at abstract level. No independent replication, participant-level reanalysis or formal quantitative risk-of-bias scoring was conducted.

3. Early evidence and the meaning of simulation

Argyle and colleagues examine whether conditioning GPT-3 on descriptions can reproduce aspects of human response distributions. Their article presents evidence of fine-grained, demographically related variation rather than treating model bias as a single population-level attribute [1]. This is an important demonstration of conditional behaviour. It leaves the accuracy of other models, prompts, topics and future survey questions to be tested.

Aher, Arriaga and Kalai propose Turing Experiments as tests of a model’s ability to simulate representative samples in human-subject research. Their framework explicitly allows the discovery of systematic distortions [11]. The appropriate question is therefore richer than whether a generated response sounds human. A simulation can reproduce one behavioural pattern and deviate on another. A benchmark needs to specify which patterns count as success and compare them with human observations.

Horton, Filippas and Manning discuss language models as simulated economic agents. Their revised abstract describes giving agents endowments, information and preferences, then examining behaviour in scenarios derived from earlier economic experiments [12]. This is a use of simulation to explore behavioural models. It does not make every model output an empirical observation of a real consumer. The distinction resembles the difference between studying the implications of a formal model and estimating the population parameters to which that model will be applied.

The early literature establishes that structured simulation can be scientifically informative. It also makes validation outcome-specific. A model that reproduces the direction of a classic treatment effect may still produce the wrong baseline level or the wrong variation between people. Those deficiencies matter for different uses: a qualitative demonstration, a power calculation and a market-size estimate require different kinds of agreement.

4. Distributional agreement can conceal inferential error

Bisbee and colleagues compare ChatGPT-generated responses with American National Election Study data. Their publisher article reports that synthetic average feeling-thermometer scores can resemble the human averages while the generated responses vary less. Regression estimates also differ from estimates based on the human data, and repeated prompts can change across collection periods [2]. The finding directly separates matching a mean from preserving statistical relationships.

The distinction is mathematical. Two datasets can share the same mean but have different variances, correlations and tails. A research question about the fraction of strongly dissatisfied consumers depends on the distribution, not just the centre. A question about how attitudes vary with age depends on a conditional relationship. Agreement on an overall average provides no general guarantee about either quantity.

Generating more responses from the same model does not necessarily resolve those errors. Additional draws can reduce Monte Carlo noise in the estimate of what the model would say under that prompt. They do not automatically reduce a systematic difference between the model’s conditional distribution and the human population. Very precise estimates of the simulated distribution can therefore coexist with poor human validity.

Synthetic observations also need an appropriate uncertainty interpretation. If many generated respondents share a model, prompt structure and source material, their variation does not represent independent sampling of human beliefs. Conventional survey standard errors can become misleading if they are attached to those outputs without a model of the prediction error. The size of the synthetic panel should not be reported as though it were the number of people interviewed.

5. Personal data and the digital-twin question

Park and colleagues build agents using self-report data from a diverse national sample of 1,052 Americans. The consulted third-version abstract distinguishes interview-grounded agents, structured-survey inputs and combinations of those sources [3]. This design asks whether detailed information about an individual can support held-out prediction across outcomes. It is more demanding than producing a demographic stereotype, but it also uses substantially more personal information.

Peng and colleagues’ revised manuscript evaluates digital twins using 19 preregistered studies with 164 outcomes. The agents receive each person’s earlier answers to more than 500 questions. The report finds only modest improvement over a homogeneous base model and an average correlation of 0.20 with human responses. It describes systematic distortions including limited individuation, stereotyping and hyper-rationality [4]. This is a preprint result, and its task set differs from the Park evaluation.

The two research programmes should not be reconciled by calling one accurate and the other inaccurate in the abstract. They use different personal inputs, target outcomes and scoring procedures. A rich interview can contain information absent from a questionnaire, while some held-out behaviours may remain difficult to predict even from many prior answers. Comparing their performance requires a common test or careful adjustment of the prediction task.

Individual prediction also has a human reliability ceiling. People can answer the same question differently on different occasions, so a model-human comparison may be normalised by a human test-retest benchmark. Such normalisation is informative when fully specified. A percentage of human repeatability is not the same as a percentage of all answers being correct. It should retain its denominator whenever quoted, particularly in commercial summaries of digital-twin accuracy.

6. Product-concept ranking and elicitation

Maier and colleagues investigate purchase-intent measurement using 57 existing consumer surveys with 9,300 responses. Their preprint compares direct numerical elicitation with a method that obtains textual responses and maps them to rating distributions using semantic similarity. The manuscript reports stronger alignment for its proposed elicitation procedure, including concept ranking relative to human test-retest reliability [5]. It is a specialised method developed and evaluated by its authors.

The design matters because a response scale is part of the measurement system. Asking a model for a number can produce a different distribution from asking for an explanation and mapping its meaning to reference statements. Improvement in that mapping demonstrates a property of the elicitation procedure on the evaluated data. It does not establish that the model has acquired the experiences, budget constraints or needs of the human respondents.

Rank agreement and level agreement have different uses. A model can order several concepts correctly while overestimating all their purchase-intent scores. That could be useful for an early screening exercise but unsuitable for forecasting demand. Conversely, similar average ratings could conceal a reversed order between close alternatives. A validation should report the target relevant to the intended decision and the uncertainty around that target.

The concept collection also defines the test domain. Results on familiar personal-care propositions may not transfer to a novel service, an unfamiliar cultural context or a product requiring physical use. A held-out validation should ensure that the evaluation concepts were not used to optimise the prompt or scoring procedure. Otherwise the measured performance can partly reflect adaptation to the test set. The review does not independently audit that boundary in the underlying datasets.

7. Newer evidence from public-opinion replication

Pew Research Center’s September 2026 report compares synthetic respondents with three American Trends Panel waves. Its tabulated average absolute percentage-point errors are 11.3, 14.7 and 11.2 for the three waves, with an average of 12.4. The specified setup uses Claude Opus 4.6 at low reasoning with extended profiles and expert reflection. The report finds substantial errors across major demographic and behavioural groups [6].

These errors concern average differences in response-option shares across questions. They are not the probability that an individual generated answer is wrong. A question with several response options contributes differences for each option under the stated aggregation rule. Interpretation requires that definition, particularly when comparing with a correlation or a concept-ranking score from another study.

The report provides a recent counterpoint to the assumption that newer models or richer profiles solve representation automatically. It remains one implementation on one public-opinion programme. Its results do not establish that every synthetic method will have the same error, but they do demonstrate that a sophisticated setup can still diverge materially from high-quality human polling. General claims of replacement need to explain such failures rather than rely on generation fluency.

8. Whose opinions appear in generated answers?

Santurkar and colleagues introduce OpinionQA to compare model responses with the views of sixty US demographic groups. Their proceedings abstract reports substantial misalignment that persists after steering models toward particular groups [7]. This concerns distributional representation in subjective questions. A model’s default answer should therefore not be interpreted as a neutral population average or an equally weighted summary of all demographic perspectives.

Durmus and colleagues’ GlobalOpinionQA research examines cross-national opinion representation. The research-team abstract reports that country prompting can move responses toward a population’s opinions while also reflecting stereotypes; translation does not necessarily make answers resemble those of speakers of the target language [8]. Specifying a country or translating a questionnaire is therefore an intervention whose validity needs checking, rather than a guarantee of cultural grounding.

Cao and colleagues similarly examine ChatGPT’s cultural alignment. Their proceedings abstract reports stronger adaptation to American context than to other cultural contexts, with English prompts reducing response variation and flattening cultural differences [9]. These findings concern the tested model and prompts. They establish the relevance of language and context as measurement conditions, without providing a timeless ranking of model cultures.

Representation matters even in apparently ordinary marketing questions. A demographic profile can activate a familiar narrative about a group while missing diversity within it. If generated responses are then interpreted as consumer evidence, the procedure risks converting model regularities into claims about people. Measuring subgroup means alone may not reveal the missing internal variation. Individual-level and distributional checks answer complementary questions.

9. Response biases, stability and validation design

Tjuatja and colleagues compare language-model responses with established human survey-response biases. Across nine models, their abstract reports that popular systems often fail to reproduce human-like patterns, especially after human-feedback training. Models can also respond to perturbations that do not produce significant changes in humans [10]. Prompt sensitivity is therefore not automatically evidence of human realism, even though people themselves are sensitive to questionnaire design.

Repeated generation provides a stability test for the implementation, but stability and validity are separate properties. A system can reproducibly give the wrong response distribution. A model update can also change an established result even when the prompt is unchanged. Version, sampling settings, system instructions and collection date belong to the definition of the simulation. Reporting only a model family name leaves important sources of variation unspecified.

A useful validation separates development from evaluation. Prompt tuning, persona design and scoring choices can use one dataset, while an untouched human dataset tests generalisation. Where several procedures are tried and only the best result is reported, the selected estimate may be optimistic. That concern applies to ordinary predictive modelling as well as synthetic respondents. A clear record of alternatives and held-out outcomes makes the claim more interpretable.

The comparator should also match the use. Predicting individual answers may be judged against a simple baseline using the person’s prior responses. Concept ranking may be judged against repeated human panels. Population estimation may require a probability-based reference and scrutiny of nonresponse. No single baseline dominates all these questions. A claimed synthetic replacement needs to state which human observation it is replacing and the loss considered acceptable.

Rubin’s causal framework distinguishes an estimate of a treatment contrast from a description of the observed sample. [13] Allcott examines site-selection bias in programme evaluation, showing why transport across settings is a separate evidential problem. [14] CheckList tests linguistic capabilities and behaviours rather than relying on one aggregate accuracy measure. [15]

Model cards document intended uses and evaluation conditions; a synthetic-panel result needs an equally explicit account of its target population and use. [16] Datasheets for Datasets provide a structured account of data collection and composition, which remain material when human records ground a simulated panel. [17] Shmueli separates explanation from prediction. Matching observed answers is consequently a different claim from recovering the mechanism behind a consumer response. [18]

D’Amour and colleagues show that similar held-out performance can coexist with different deployment behaviour. Agreement on one validation sample cannot establish interchangeability in every downstream study. [19]

10. Synthesis and conclusion

The selected literature establishes conditional successes in behavioural simulation and concept evaluation, alongside substantial failures in distributional representation and individual prediction. Rich personal grounding and specialised elicitation can change performance, but they do not produce a common guarantee of human validity. Agreement with one reference statistic cannot validate all the other quantities that a research user might infer.

The review’s source access and selection limit its conclusions. Several papers were consulted only at abstract level, and preprints have not been treated as settled standards. No independent analysis of the original response data was conducted. The studies also differ in model version, personal information, outcome domain and evaluation metric. These differences prevent a pooled replacement rate and leave many commercial settings untested.

Synthetic responses are best interpreted as model predictions whose evidential value depends on a specified validation target. A large panel, realistic language or accurate overall mean is insufficient on its own. The empirical question is whether the procedure preserves the response distribution, subgroup relationship or individual outcome required by the intended use, on data that were not used to construct the procedure. Existing research supplies methods for asking that question and evidence that the answer varies across tasks.

Source access and appraisal register

The additional comparative sources in references 13–19 are used for the specific distinctions cited in the discussion. Access is limited to previously inspected primary-source abstracts, publication records or manuscripts. Their inclusion does not imply a fresh full-text risk-of-bias appraisal or direct validation in the application discussed here.

The register describes the material consulted for this narrative review. Access to a manuscript does not imply replication, independent extraction or a formal risk-of-bias assessment. Source dates identify the underlying research.

References