MixedAudit of an AI system

Add a note

Analytic flexibility in silicon samples: re-examining Argyle et al. Study 3

What was attempted

Silicon samples are LLM-generated datasets meant to stand in for human respondents. In Study 1 the author generated 252 silicon-sample configurations (varying model, sampling parameters, prompt format and demographic detail) for a controlled case using two social-psychological scales. In Study 2 he re-examined Study 3 of Argyle et al. (2023), a published silicon-sample use case, with 66 alternative configurations. The question was how far defensible choices change conclusions about how well silicon samples match human data.

What happened

  • Study 1: configurations varied substantially on all three criteria (recovering participant rankings, response distributions and between-scale correlations), and configurations that did well on one often did poorly on another.
  • Study 2: the correlation between human and silicon association structures ranged from r = .23 to r = .84 across the 66 configurations.

Why it failed or succeeded, per the authors

  • Different defensible configuration choices can change conclusions about fidelity, so a single reported configuration may mislead.
  • The author states that "different defensible configuration choices can materially alter conclusions about the fidelity of silicon samples".
  • He calls for greater attention to analytic flexibility and outlines strategies researchers can use to reduce the threat.

Checks used

  • Systematic variation of configurations against fixed human data.
  • Three fidelity criteria for Study 1; association-structure correlation for Study 2.

Limits of this case

A methods audit of one published use case and one controlled case, not a replication of a human study; the absolute range depends on the configurations the author chose. Only the abstract and introduction were read closely; the numbers above are the abstract's.

Search the collection

↑ ↓ move · Enter open · Shift+Enter original source · Esc close · Search index for agents

Keyboard shortcuts

Single keys work when you are not typing and no dialog is open. Turn them off if stray key presses move the page. ⌘K / Ctrl+K and Esc keep working.

/
Search the collection
⌘K / Ctrl+K
Open or close search
?
Show these shortcuts
h
Home
l
Lessons
a
Accounts
r
Replications
p
Papers
g
Guides and tools
d
Timeline
b
Bookmark the record you are reading (press again to remove it)
s
Your saved list and notes
← →
Previous and next record on an account, lesson, replication or paper page, and in a quick look
m
Switch between system, light and dark theme
t
Back to the top of the page
Open full page