Are Synthetic Respondents Accurate? 2026 Evidence & Limits

Synthetic respondents are AI-generated personas or simulated survey participants that produce answers intended to represent how real people might respond to research questions. They can be generated from general-purpose large language models, historical survey results, customer data, demographic profiles, or detailed individual-level behavioral information.

Unlike traditional survey respondents, synthetic respondents do not independently provide current, lived opinions. Their answers are generated from model patterns and the information supplied to the system. The result is a prediction or simulation of human responses, not a direct measurement of what real people currently think, feel, or intend to do.

The term covers several different approaches, including simple LLM personas, segment-level simulations, synthetic records used to augment existing datasets, and digital twins built from detailed personal data. These methods differ substantially in their inputs, validation requirements, costs, and potential accuracy.

For that reason, researchers should not treat all synthetic respondent products as equivalent. To evaluate a vendor's accuracy claim, they need to understand how the synthetic respondents are generated, what human data informs them, and how their outputs perform against an independent human benchmark.

Four types of synthetic respondents

Not all synthetic methods are equal. It helps to understand what you are actually getting:

1. Ungrounded LLM personas. A general-purpose AI is prompted to act like a type of person. This is the cheapest and least reliable method.

2. Segment personas. AI conditioned on segment descriptions or past survey data. Better grounded, but still predicting from historical patterns.

3. Synthetic boosters. Modeled records used to fill gaps or boost underrepresented cells in an existing dataset. Useful in narrow contexts with strong validation.

4. Digital twins. Models grounded in rich, individual-level data. Stanford HAI reported that generative agents built from extensive personal data replicated participants’ survey responses 85% as accurately as participants replicated their own answers two weeks later. But this required far more data per person than most commercial applications provide.

The accuracy of synthetic respondents varies dramatically across these methods. A vendor claiming “90% accuracy” using digital twins grounded in deep behavioral data is making a fundamentally different claim than one using prompted LLM personas.

Synthetic respondents can be accurate enough for low-stakes, directional tasks like brainstorming, survey logic testing, exploring hypotheses, or early concept screening. They are not consistently accurate enough for decision-grade research.

Here is a practical breakdown:

The pattern is clear. Synthetic respondents are useful before fieldwork. They are risky instead of fieldwork.

Independent research suggests that synthetic respondent accuracy depends on the model, the data used to build it, the research task, and the metric used to evaluate it. Matching an overall survey result does not necessarily mean a model can reproduce subgroup differences, identify purchase drivers, or predict actual consumer behavior.

These findings are not directly comparable. The studies use different models, datasets, benchmarks, and definitions of accuracy. They should be interpreted as evidence about specific evaluations, not as a universal ranking of synthetic respondent technologies.

Research sources: Pew Research Center, Burke, STRAT7, and Stanford HAI.

A synthetic respondent accuracy claim is meaningful only when the vendor explains what was measured, what the model was compared against, and how the result was calculated. A reported accuracy of 90% could refer to matching individual answers, reproducing overall response distributions, identifying differences between audience segments, or predicting real-world behavior. These are different performance measures and cannot be treated as interchangeable.

For market research, the most important question is not whether synthetic data looks similar to human data on average. It is whether the model preserves the specific information needed to make a decision. A small error in an overall percentage might be acceptable for early exploration but unacceptable if it reverses a product ranking, changes a pricing recommendation, or conceals a meaningful difference between customer groups.

Researchers should therefore evaluate synthetic respondents at the level where the decision will be made. A vendor that reports aggregate agreement should not imply that the model has also demonstrated respondent-level accuracy, reliable subgroup estimates, valid purchase drivers, or predictive performance.

Accurate Against What Benchmark?

An accuracy claim may measure one or more of the following:

• Topline agreement: Whether overall response percentages resemble those from human respondents.

• Individual-level agreement: Whether synthetic answers match the answers of specific human participants.

• Subgroup accuracy: Whether results preserve differences between demographic or customer groups.

• Relationship preservation: Whether correlations, purchase drivers, and other relationships between variables remain similar.

• Behavioral prediction: Whether synthetic results predict observable outcomes, such as actual purchases or product choices.

A strong result on one measure does not establish strong performance on the others. Ask vendors to report the metric, validation dataset, uncertainty, and intended use case behind every headline accuracy figure.

Vendors frequently cite accuracy numbers like 80%, 86%, or 90%. These numbers are not meaningless, but they are often incomplete in ways that matter for business decisions.

Many researchers are skeptical of these claims for the same reason. A “90% accurate” figure only tells you something if you know what it’s being measured against. A model that closely matches what people say in a survey isn’t necessarily predicting what they’ll actually do, like whether they’ll buy the product.

Accurate against what?

An accuracy number could mean the synthetic output matches:

• Historical survey responses from a public dataset

• A holdout sample of human answers on the same survey

• Overall distribution of responses (toplines)

• Subgroup-level patterns (crosstabs)

• Relationships between variables (drivers)

• Actual purchasing or behavioral data

Each of these benchmarks tells you something different. Matching toplines is the easiest test. Preserving driver relationships and subgroup differences is much harder. Predicting actual behavior is harder still.

The 80% accuracy trap

Burke’s research illuminates this problem. At a commonly cited 80% accuracy level, LLM-based synthetic data still produced false conclusions in roughly 60% of tested business scenarios. The reason: business decisions rarely depend on whether the overall average is close. They depend on which option wins, which segment differs, which driver matters most. A model can get the average right and still point you toward the wrong decision.

A 2026 paper on arXiv (“Synthetic Data in Marketing Research: How to Evaluate and When to Trust”) makes a similar point: aggregate metrics can look acceptable while masking a lack of respondent-level differentiation.

Synthetic sample size is not human sample size

Another common misconception is that a larger synthetic sample improves accuracy the way a larger human sample does. It doesn’t, because every synthetic response comes from the same model and the same assumptions. Adding more of them doesn’t add new information. It just repeats the model’s view more times.

A synthetic sample of 5,000 does not carry the same statistical meaning as a human sample of 5,000. The additional outputs tighten the illusion of precision while systematic bias remains unchanged.

Synthetic respondents are most useful when they help researchers prepare, refine, or prioritize human research rather than substitute for it. They can help teams test survey wording, identify missing response options, generate hypotheses, and narrow a large set of concepts before investing in human fieldwork.

The value comes from accelerating the research process, not from assuming that generated answers are equivalent to measured human opinions. Each use case should have a clear purpose, defined limitations, and a validation step whenever the output could influence a consequential decision.

Pretesting survey instruments. Before spending money on human fieldwork, synthetic respondents can identify confusing question wording, check routing logic, pressure-test response options, and flag edge cases. This is one of the strongest use cases because the synthetic output is improving the research instrument, not replacing the evidence.

Hypothesis generation. Synthetic respondents can help teams explore possible barriers, motivations, and objections before designing a study. Think of this as structured brainstorming with a demographic lens.

Early concept iteration. If a team has 30 possible claims and budget to test only 8 with real people, synthetic respondents can help narrow the list. The final test should still use verified human respondents, but the narrowing step is a reasonable application.

Data augmentation with strong validation. When synthetic records are used to fill gaps in underrepresented cells of an existing dataset, and when that augmentation is rigorously validated against holdout human data, the method can add value. This is a technical use case that requires expertise and governance.

In practice, synthetic results often match humans on general patterns, but not as closely as teams expect. Synthetic users may help with early iterations, but they cannot explain why they answered as they did in a research-valid way. That is the core distinction: synthetic respondents can produce answers, but those answers are generated rationalizations, not lived causes.

Synthetic respondents are strongest when they support research operations. They are weakest when they replace measurement.

Synthetic respondents become harder to trust when research depends on precise subgroup differences, real purchasing behavior, changes over time, or the discovery of unexpected human motivations. These tasks require more than plausible answers or close agreement with an overall survey average.

The following limitations matter most when synthetic results could influence pricing, segmentation, brand strategy, product launches, or public-facing claims. The strength of the evidence varies by use case, so individual findings should be interpreted in the context of the study that produced them.

Subgroup analysis

Pew found materially higher errors for certain demographic groups: 16.1 percentage points for Republican or Republican-leaning adults and 15.1 points for Black adults. In one striking example, synthetic polls predicted that 97% of Hispanic adults were at least somewhat likely to follow the World Cup when the actual share was 42%. That is not directional error. That is stereotype amplification.

General-purpose AI models are often not sensitive enough for meaningful subgroup analysis. STRAT7’s benchmark found a 17-point miss for women aged 35-44 on a single snack-buying question. If your research depends on understanding differences between audience segments, synthetic respondents are not reliable enough without human validation.

Pricing research

STRAT7 found that pure synthetic respondents contradicted their own price ordering 68% of the time in a pricing exercise. This means the model could not even maintain internal consistency on willingness to pay, let alone predict actual price sensitivity in market conditions.

Brand tracking

Tracking change over time is one of the hardest tasks for synthetic respondents. STRAT7 reported that synthetic data correctly tracked year-on-year brand awareness movement only 19% of the time. For companies running ongoing brand perception studies, this makes synthetic tracking unreliable as a standalone method.

Driver analysis and segmentation

Getting the averages right isn’t enough. The real test is whether synthetic data keeps the relationships in the data intact, like which factors actually drive a purchase. That’s where it most often falls short. Researchers at Vanderbilt found that synthetic responses matched real survey averages closely, but the relationships between variables often came out significantly different. For example, in one STRAT7 test, a provider’s driver analysis ranked store availability as a top driver, when real shoppers ranked it sixth.

Novel products and emerging behavior

New products are especially hard for synthetic respondents. A model only knows what’s in its training data, so anything genuinely new may be out of reach. Even digital twins struggle. A Columbia Business School study built twins from each person’s answers to more than 500 questions, then ran 19 studies. The twins’ answers correlated only modestly with the real people’s (about 0.2 on average) and were less varied. Synthetic respondents are weakest where market research is most valuable, which is discovering what the team did not already know.

Emotion and embodied experience

Synthetic respondents tend to repeat familiar language. In a Nielsen Norman Group study, human respondents used more than 50 distinct terms to describe their work, while synthetic respondents stuck to a much narrower, predictable set. They also fall short where emotion and lived experience matter. As Greenbook points out, real interviews surface uncertainty, humor, and social influence that researchers never anticipated.

Synthetic output often reads cleanly, which is actually part of the problem. Real human responses are messy, contradictory, and surprising. That messiness is signal, not noise.

Public-facing claims

If a brand wants to publish “62% of Gen Z consumers prefer X” based on synthetic respondents, that should be avoided or heavily disclosed. AAPOR’s 2026 ethics code states that terms like “survey” and “poll” imply human respondents as the primary data source. Presenting synthetic output as consumer survey data creates both ethical and reputational risk.

For teams that need credible, publishable research findings, the data should come from real people. Research-led content programs built on proprietary human data are far more defensible than synthetic outputs dressed up as consumer insight.

The core difference comes down to a simple framing: synthetic respondents are predictions, and human respondents are measurements. Predictions can help you think faster. Measurements are what you need when the answer has to represent real people right now.

Synthetic respondents can guess what someone like your customer might say. Market research needs to know what a real customer does when they see your product, right now.

That gap shows up in testing. Researchers at Columbia Business School found that when you change one thing in a synthetic test, like the price, the AI quietly changes other parts of the scenario too. Compared against a real experiment with 40 products, the results didn’t hold up. One is a guess. The other is a measurement.

The Nuremberg Institute for Market Decisions found that synthetic respondents matched real participants’ choices about 79% of the time in a brand-funnel setting, but overestimated positive attitudes toward well-known brands and produced less variation than real participants. Even in the best case, synthetic output tends to smooth over the real diversity of human opinion.

User Interviews surveyed 150 researchers in 2026 and found 47% described themselves as skeptical and wanting more evidence before trusting synthetic users, while only 24% were cautiously optimistic. The research community itself has not yet reached confidence in synthetic respondent accuracy.

If a vendor says their synthetic respondents are accurate, ask these questions before trusting the claim:

1. What is the synthetic respondent conditioned on? A generic LLM prompt produces very different reliability than a model grounded in extensive individual-level behavioral data. Ask what data feeds the simulation.

2. What does the accuracy number measure? Topline overlap? Individual answer matching? Segment-level match? Driver preservation? Behavioral prediction? The metric matters more than the number.

3. What is the benchmark? A holdout human sample is the strongest benchmark. Past surveys, public datasets, or internal validation sets are weaker. Actual purchase data would be strongest of all, but is rarely used.

4. How did it perform for subgroups? Ask for error rates by age, gender, income, ethnicity, buyer type, and any audience critical to your decision. Overall accuracy can hide large subgroup failures.

5. Does the model preserve relationships between variables? Ask whether it preserves drivers, crosstabs, segment differences, and rank order, not just totals.

6. Can the vendor show failure cases? Vendors with credible validation should be able to tell you where their method fails. If they cannot, the validation is incomplete.

7. How is AI or synthetic use disclosed? The ICC/ESOMAR 2025 Code requires clients to be informed when synthetic data or synthetic personas are used, and the extent of human oversight must be stated.

8. Would you make the same decision if this answer were wrong? If not, validate with real humans.

Red flags in vendor claims

Watch for these warning signs: “90% accurate” with no benchmark specified, no holdout human validation, no subgroup error reporting, treating synthetic sample size like human sample size, claiming margin of error for a fully synthetic sample, or using terms like “respondents” without clearly stating they are AI-generated.

Green flags in vendor claims

Better vendors will provide side-by-side validation against human holdout samples, report error by subgroup, separate topline accuracy from decision reliability, state where the model should not be used, and keep humans in the loop for high-stakes interpretation.

The practical rule of thumb is straightforward: use synthetic respondents when the cost of being wrong is low and the goal is to improve the research process. Use verified human respondents when the cost of being wrong is high and the goal is to measure the market.

Companies like Helcim have found that relying on internal data alone creates an echo chamber. Getting external market perspective from real people breaks that cycle. Similarly, Greenhouse used real human research to validate what was signal versus industry noise, which is exactly the kind of ground-truthing that synthetic respondents cannot reliably provide.

There is a strong case for using AI in market research, just not as a replacement for real people. AI is valuable when it accelerates study design, analysis, synthesis, and reporting while keeping verified human respondents as the evidence base. That combination gives you speed without sacrificing validity.

See how Cashew’s pricing model scales respondent sourcing and support with your study needs.

The question “are synthetic respondents accurate?” is incomplete without specifying accurate for what decision, against what benchmark, and at what level of analysis. Here is what the independent evidence tells us:

• Synthetic respondents can match broad toplines on familiar, well-documented topics, but they break down on subgroups, drivers, pricing, tracking, and novel behavior.

• An “80% accurate” synthetic dataset can still produce false conclusions in the majority of business scenarios.

• Bigger synthetic samples do not fix model bias the way bigger human samples reduce sampling error.

• Synthetic respondents amplify stereotypes and smooth over the real diversity of human opinion.

• Industry standards from AAPOR and ICC/ESOMAR require clear disclosure when synthetic data is used, and they draw a firm line between AI-generated outputs and human research participants.

• The strongest use of synthetic respondents is as a pre-research tool that improves instruments and generates hypotheses, not as a replacement for human measurement.

Synthetic respondents are getting better, but they are not a replacement for verified human research. They can help teams think faster, test assumptions, and improve research design. When the answer needs to reflect real people, real context, current behavior, or a decision that matters, the safest path is still to ask verified humans.

For teams building evidence-based marketing programs or making go/no-go decisions on new products, the stakes are too high for simulated answers.

Talk to Cashew about verified human research.

Are synthetic respondents real people?

No. Synthetic respondents are AI-generated or model-generated responses, not human research participants. AAPOR explicitly distinguishes AI-generated responses from research participants and requires they be identified as AI-created if included in a study of public opinion.

Can synthetic respondents replace survey panels?

Not for decision-grade research. They can support early-stage exploration, survey QA, and hypothesis generation, but they should not replace human panels for validation, tracking, segmentation, pricing, or public claims. Independent benchmarks consistently show that synthetic outputs degrade as the analysis gets more specific.

What does “90% accurate” mean for synthetic respondents?

It depends entirely on the benchmark. It might mean aggregate distribution overlap with a past survey, individual answer matching on a holdout set, or correlation with a validation dataset. The number is not meaningful unless the method, benchmark, level of analysis, and use case are all disclosed. As Burke’s research shows, even 80% accuracy can lead to wrong conclusions.

Are synthetic respondents good for concept testing?

They can help narrow a long list of ideas before fieldwork, which saves time and budget. But final concept validation should use real respondents. Synthetic output tends to overestimate positive reactions to familiar concepts and lacks the ability to capture genuine surprise or emotional response to novel ideas.

Are synthetic respondents good for pricing research?

Current evidence suggests high risk. STRAT7 found pure synthetic respondents contradicted their own price ordering 68% of the time. Pricing decisions depend on real willingness-to-pay in specific market contexts, which synthetic models cannot reliably simulate.

Can synthetic respondents be used in published research?

Only with clear and prominent disclosure. AAPOR and ICC/ESOMAR both require transparency about AI and synthetic data use. Presenting synthetic outputs as if they came from surveyed consumers is misleading and creates reputational risk. Brands that want credible, publishable research should ground their claims in data from real people.

Why do synthetic respondents struggle with subgroups?

Large language models are trained on internet-scale text data, which encodes dominant narratives and stereotypes about demographic groups. When prompted to answer “as” a specific type of person, they tend to produce responses that reflect cultural assumptions rather than the actual diversity of opinion within that group. Pew’s finding that synthetic polls predicted 97% World Cup interest among Hispanic adults (versus 42% actual) illustrates this problem clearly.

How should synthetic respondents be labeled in reports?

Industry standards require that any use of AI-generated data, synthetic personas, or synthetic sample be disclosed to clients and stakeholders. The extent of human oversight should be stated. Reports should never use terms like “survey” or “poll” to describe synthetic output without making the methodology explicit.