Partly succeededReplication study

Add a note

Reproducing 48 social-science papers from the methods description alone

What was attempted

The authors asked whether agents can reproduce results without the original code. A system extracts a structured description of each paper's methods, masks the reported numbers, and gives an agent only that description and the original data. Agents never see the original code, results or paper. Four agent scaffolds and four LLMs were run on 48 papers that the Institute for Replication had verified as reproducible. Outputs were compared cell by cell with the originals, and an LLM traced each discrepancy to a root cause.

What happened

  • Three best agents matched the sign of the original coefficient about 85 to 91% of the time.
  • The best agent (GPT-5.4 on OpenCode) was within 1.96 original standard errors over 80% of the time; it beat Claude Code (Opus 4.6), GPT-5.3 Codex and GLM-5.
  • Scaffold mattered: GPT-5.4 on OpenCode beat GPT-5.4 on Codex CLI and on mini-SWE-Agent, with much greater token use.
  • Root-cause labels: most divergences were attributed to paper under-specification, fewer to agent errors, with the share of agent errors lower for stronger agents.

Why it failed or succeeded, per the authors

  • Errors come mostly from papers under-specifying their methods and, to a lesser extent, from agents misreading or ignoring the method description. The abstract says "agents can largely recover published results".
  • Better results came with more token use and longer analysis runs.

Checks used

  • Strict information isolation between the agent and the original code and results.
  • Deterministic, cell-level comparison against the published tables.
  • A human audit of the LLM root-cause step: of 53 flagged divergences, 49 confirmed as meaningful (92%), but only 33 of 49 error-source labels confirmed (67%), 10 unsure. A different LLM judge changed the agent-error share.

Limits of this case

Papers were pre-selected as verified-reproducible and the data were given, so this tests re-implementation, not discovery of data. Root-cause shares come from an LLM judge and are indicative only, as the authors say.

Search the collection

↑ ↓ move · Enter open · Shift+Enter original source · Esc close · Search index for agents

Keyboard shortcuts

Single keys work when you are not typing and no dialog is open. Turn them off if stray key presses move the page. ⌘K / Ctrl+K and Esc keep working.

/
Search the collection
⌘K / Ctrl+K
Open or close search
?
Show these shortcuts
h
Home
l
Lessons
a
Accounts
r
Replications
p
Papers
g
Guides and tools
d
Timeline
b
Bookmark the record you are reading (press again to remove it)
s
Your saved list and notes
← →
Previous and next record on an account, lesson, replication or paper page, and in a quick look
m
Switch between system, light and dark theme
t
Back to the top of the page
Open full page