Foundation Model Limitations In Reasoning About Humans: When AI Explains People, Plausibility Outruns Evidence

Casey Jones

When an AI is asked how a population will react to a policy, why a consumer will buy a product, why a soldier will follow or refuse an order, or why an employee or voter is quietly disengaging, it will always produce an explanation before it has established that the explanation is true.

The problem is that the accounts generated will always be coherent, detailed, and psychologically recognizable; yet these qualities do not establish that the system has correctly inferred anyone's beliefs, identified the motive behind their behaviors, or represented the population whose response it is predicting.

The gap between an explanation and a model of a person

Human statements are evidence about human beings. They are downstream of the forces that shape them. Their meaning depends on circumstances, incentives, relationships, and what a person is able or willing to articulate.

A declaration of agreement can coexist with reluctance. Silence can be consistent with approval, fear, resignation, or withdrawal. A purchase can reflect aspiration, necessity, habit, or restricted alternatives. It is an often ignored fact that several explanations can fit the same visible act while implying different reactions to what happens next.

Even people's explanations of their own decisions are known to be unreliable. Choice-blindness experiments have shown that participants often supplied reasons for choices they had not actually made, after the apparent outcome was covertly changed. An articulate explanation is quite frequently an individual's reconstruction rather than an accurate report of the cognitive process that produced the decision.¹

Observed behavior requires interpretation as well. A system that predicts what someone will click has not necessarily identified what that person values. The desired mental state and the measured action can diverge.²

The challenge is to distinguish competing explanations and determine which relationships continue to hold when circumstances change.

Research has identified failures that fluency can conceal

ExploreToM³ demonstrated that familiar social reasoning performance can coexist with severe failures on adversarial belief-tracking tests. Those results expose weaknesses in generalization; they do not measure the failure rate of every real-world social prediction.

A 2026 study of economic games found that models could approach human performance in predicting actions while failing to transfer inferred motives to a different game.⁴ This is an especially consequential gap: a model can appear to understand an actor until a change in circumstances requires that understanding to do useful work.

Population simulations introduce further problems. Research has found that demographic personas can reproduce outsiders' impressions of groups and suppress variation within them.⁵ Other work has shown that plausible averages can coexist with incorrect relationships between demographic characteristics and opinions.⁶

An October 2026 comparison spanning 74 countries found that simulated economic preferences captured some patterns but remained weak on cultural differences and broader policy inference, scoring only 48.1% on cultural dimensions.⁷

These failures matter because decisions depend on more than the average response. For illustration, two populations could report the same average support for a policy while one contains mostly mild agreement and the other contains sharply opposed factions. Their aggregate score would not establish that they would respond similarly to implementation.

A collection of generated personas becomes a useful population model only when its variation, composition, and relevant relationships are supported by evidence.

The problems persist even when a model has access to the relevant research

A 2026 study found that language models will programmatically strengthen correlational findings into causal claims, particularly when asked to simplify the material or emphasize practical implications.⁸ Providing the underlying AI with full articles did not reliably remove that tendency; although providing it with instructions emphasizing methodological care in its responses helped, but only if a user takes the time to prepend their prompts carefully.

This creates a practical risk. A source may establish an association. The generated summary may describe a cause. A subsequent recommendation may then assume that changing the supposed cause will change the outcome. Each step can sound reasonable even though the evidence does not support the final intervention.

The questioner's own assumptions also enter the answer. Research on sycophancy shows that models favor agreement with users⁹, and experiments have found that such interactions will strengthen people's conviction that they are right while reducing their expressed willingness to repair interpersonal conflict.¹⁰

An explanation of another person should therefore be examined for evidence of that person's circumstances and perspective, as well as for evidence that the system has accepted the questioner's framing.

Why better retrieval is only part of the remedy

Retrieval can improve an analysis by supplying observations that were missing: behavioral records, interviews, experiments, representative surveys, and relevant history.

Its contribution depends on how those materials are used. Retrieving a person's words does not determine whether they express a stable preference, a temporary intention, a social obligation, or a justification. Retrieving behavior does not uniquely identify the motive behind it. Retrieving a study does not establish that its findings transfer to a different population.

The underlying problem is one of inference. Which explanation is supported? What alternatives remain? What new evidence would distinguish them? Under which conditions should the prediction continue to hold?

The fact that a motive is unspoken does not make it inaccessible to investigation. It makes the quality of the inference especially important. Hidden motives must remain hypotheses until evidence justifies stronger claims.

What progress would require

A system used to reason about people should distinguish observations from inferred motives, represent meaningful differences within the population, and test whether its predictions hold for new people, later periods, and changed circumstances. Claims about causes require evidence that distinguishes those causes from plausible alternatives.

The central research task is to establish reliable relationships among what people say, what they do, the circumstances in which they act, and the motives inferred from those observations.

A system that explains people must be judged by what its explanations enable it to predict, distinguish, and withstand—not by how readily those explanations persuade a reader.

References

1. Johansson, P., Hall, L., Sikström, S., & Olsson, A. (2005). Failure to detect mismatches between intention and outcome in a simple decision task. Science, 310(5745), 116–119. doi:10.1126/science.1111709. ↩
2. Kleinberg, J., Ludwig, J., Mullainathan, S., & Raghavan, M. (2024). The Inversion Problem: Why Algorithms Should Infer Mental State and Not Just Predict Behavior. Perspectives on Psychological Science, 19(5), 827–838. doi:10.1177/17456916231212138. ↩
3. Sclar, Melanie, et al. (2025). Explore theory of mind: program-guided adversarial data generation for theory of mind reasoning. International Conference on Learning Representations, Vol. 2025. ↩
4. Can Large Language Models Infer Human Actions and Motives? Evaluation in Social Prediction and Inspection Games. (2026). In Findings of the Association for Computational Linguistics: ACL 2026 (pp. 32791–32803). San Diego, California: Association for Computational Linguistics. ↩
5. Wang, A., Morgenstern, J., & Dickerson, J. P. (2025). Large language models that replace human participants can harmfully misportray and flatten identity groups. Nature Machine Intelligence, 7, 400–411. doi:10.1038/s42256-025-00986-z. ↩
6. Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B., & Larson, J. M. (2024). Synthetic Replacements for Human Survey Data? The Perils of Large Language Models. Political Analysis, 32(4), 401–416. doi:10.1017/pan.2024.5. ↩
7. Chen, F., Wang, Q., Wang, L., & Bhatia, S. (2026). Simulated responses from large language models are weak proxies for human economic preferences across 74 countries. Nature Human Behavior. Advance online publication, October 12, 2026. doi:10.1038/s41562-026-02611-4. PMID: 42864430. ↩
8. Isch, C., Dörr, T., Fasching, N., et al. (2026). Quantifying the prevalence and impact of overreaching causal claims in social science. Nature Human Behavior. doi:10.1038/s41562-026-02553-x. ↩
9. Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S., Durmus, E., Hatfield-Dodds, Z., Johnston, S., Kravec, S., & Maxwell, T. (2024). Towards understanding sycophancy in language models. In International Conference on Learning Representations (Vol. 2024, pp. 110–144). arXiv:2310.13548. ↩
10. Cheng, M., Lee, C., Khadpe, P., Yu, S., Han, D., & Jurafsky, D. (2026). Sycophantic AI decreases prosocial intentions and promotes dependence. Science, 391(6792), eaec8352. doi:10.1126/science.aec8352. PMID: 41886588. ↩