MixedReplication game

Add a note

Institute for Replication AI replication games: human-only, AI-assisted and AI-led teams

What was attempted

In randomised replication games, 288 researchers formed 103 teams in three arms: human-only, AI-assisted (ChatGPT as a collaborator) and AI-led (ChatGPT drives, with minimal human oversight). Each team had seven hours to computationally reproduce a few pre-selected results from a published paper, find coding errors, and propose robustness checks. Twelve studies were used across events. Some papers contained coding errors the organisers had found earlier and not published. The published date above is the compile date of the accepted PNAS manuscript, not the journal issue date.

What happened

  • Reproduction rate: human-only 94% (31 of 33 teams), AI-assisted 91% (32 of 35), AI-led 37% (13 of 35).
  • Major coding errors found per team: human-only 1.70, AI-assisted 0.74, AI-led 0.23.
  • Minor errors: 1.00, 1.40 and 0.69; differences were not statistically significant.
  • AI-led teams proposed weaker robustness checks, six supplied none rated as good, and took longer.

Why it failed or succeeded, per the authors

  • Participants in AI-led teams reported prompt fatigue, hallucinated reasoning paths, truncated context windows and long debugging loops.
  • AI-assisted teams outsourced routine tasks such as boilerplate code and file paths, and kept conceptual control.
  • The authors say the AI-led arm is probably a lower bound, since better prompting, newer models and supervision could raise it.

Checks used

  • Randomised assignment to arms, with a one-hour ChatGPT training for the AI arms.
  • Reproduction counted only if the team reproduced the pre-selected results.
  • Transcripts of ChatGPT use retained, plus focus-group material for explanation.
  • The authors conclude that "expert human judgment currently remains indispensable for reliable empirical verification".

Limits of this case

Only ChatGPT, only seven hours, and a small non-random set of twelve studies. The models used were 2024-era GPT-4 and GPT-4o; later agentic tools do much better on similar tasks (see socsci-repro-bench.md).

Search the collection

↑ ↓ move · Enter open · Shift+Enter original source · Esc close · Search index for agents

Keyboard shortcuts

Single keys work when you are not typing and no dialog is open. Turn them off if stray key presses move the page. ⌘K / Ctrl+K and Esc keep working.

/
Search the collection
⌘K / Ctrl+K
Open or close search
?
Show these shortcuts
h
Home
l
Lessons
a
Accounts
r
Replications
p
Papers
g
Guides and tools
d
Timeline
b
Bookmark the record you are reading (press again to remove it)
s
Your saved list and notes
← →
Previous and next record on an account, lesson, replication or paper page, and in a quick look
m
Switch between system, light and dark theme
t
Back to the top of the page
Open full page