# Analytic flexibility in silicon samples: re-examining Argyle et al. Study 3

- Outcome: Mixed
- By: Jamie Cummins and colleagues
- Field: social psychology / computational social science
- Published: 2025-09-16
- Source: https://arxiv.org/abs/2509.13397

## What was attempted

Silicon samples are LLM-generated datasets meant to stand in for human respondents. In Study 1 the author generated 252 silicon-sample configurations (varying model, sampling parameters, prompt format and demographic detail) for a controlled case using two social-psychological scales. In Study 2 he re-examined Study 3 of Argyle et al. (2023), a published silicon-sample use case, with 66 alternative configurations. The question was how far defensible choices change conclusions about how well silicon samples match human data.

## What happened

- Study 1: configurations varied substantially on all three criteria (recovering participant rankings, response distributions and between-scale correlations), and configurations that did well on one often did poorly on another.
- Study 2: the correlation between human and silicon association structures ranged from r = .23 to r = .84 across the 66 configurations.

## Why it failed or succeeded, per the authors

- Different defensible configuration choices can change conclusions about fidelity, so a single reported configuration may mislead.
- The author states that "different defensible configuration choices can materially alter conclusions about the fidelity of silicon samples".
- He calls for greater attention to analytic flexibility and outlines strategies researchers can use to reduce the threat.

## Checks used

- Systematic variation of configurations against fixed human data.
- Three fidelity criteria for Study 1; association-structure correlation for Study 2.

## Limits of this case

A methods audit of one published use case and one controlled case, not a replication of a human study; the absolute range depends on the configurations the author chose. Only the abstract and introduction were read closely; the numbers above are the abstract's.
