MixedAudit of an AI system
Analytic flexibility in silicon samples: re-examining Argyle et al. Study 3
Also mixed
- LLM workflow run over 4,452 economics replication packages1 Sep 2026
- Coding agents replicating four scientific machine learning papers with an evidence workflow2 Jul 2026
- SocSci-Repro-Bench: Claude Code and Codex reproducing 54 social-science papers9 Jun 2026
- Institute for Replication AI replication games: human-only, AI-assisted and AI-led teams26 Mar 2026
- Auditing three Kosmos radiation-biology claims against null models17 Nov 2025
- Kosmos reproducing unpublished and preprinted findings (developer-run)4 Nov 2025
What was attempted
Silicon samples are LLM-generated datasets meant to stand in for human respondents. In Study 1 the author generated 252 silicon-sample configurations (varying model, sampling parameters, prompt format and demographic detail) for a controlled case using two social-psychological scales. In Study 2 he re-examined Study 3 of Argyle et al. (2023), a published silicon-sample use case, with 66 alternative configurations. The question was how far defensible choices change conclusions about how well silicon samples match human data.
What happened
- Study 1: configurations varied substantially on all three criteria (recovering participant rankings, response distributions and between-scale correlations), and configurations that did well on one often did poorly on another.
- Study 2: the correlation between human and silicon association structures ranged from r = .23 to r = .84 across the 66 configurations.
Why it failed or succeeded, per the authors
- Different defensible configuration choices can change conclusions about fidelity, so a single reported configuration may mislead.
- The author states that "different defensible configuration choices can materially alter conclusions about the fidelity of silicon samples".
- He calls for greater attention to analytic flexibility and outlines strategies researchers can use to reduce the threat.
Checks used
- Systematic variation of configurations against fixed human data.
- Three fidelity criteria for Study 1; association-structure correlation for Study 2.
Limits of this case
A methods audit of one published use case and one controlled case, not a replication of a human study; the absolute range depends on the configurations the author chose. Only the abstract and introduction were read closely; the numbers above are the abstract's.