Partly succeededBenchmark

Add a note

ReplicatorBench: agents replicating social and behavioural science claims with new data

What was attempted

Unlike code-rerun benchmarks, this one asks agents to replicate a claim: find and retrieve new data, design the analysis, run it, and judge whether the claim replicates. It has 19 observational studies from the Center for Open Science's SCORE project, including human-verified non-replicable ones. Agents are scored on about 1,568 rubric checkpoints across extraction and data retrieval, design and execution, and interpretation. The authors tested their ReplicatorAgent with four LLMs and varied the programming language and level of code access.

What happened

  • Judging replicable versus non-replicable (version 3 numbers): GPT-5 accuracy 76.9% (macro F1 76.9), GPT-4o 61.5%, GPT-5-mini 48.7% (macro F1 47.9).
  • Data retrieval was the weak stage: macro F1 about 8 to 19 across settings, and the agent found all of a study's data sources (hit@all) in only 5 to 16% of studies.
  • GPT-5 and o3 had far fewer input-data errors than GPT-4o in execution (6 against 211 for GPT-5 and GPT-4o).

Why it failed or succeeded, per the authors

  • Agents "struggle with retrieving new data, necessary to replicate a claim", while designing and running analyses went well.
  • GPT-5-mini failed more on syntax and library use than on data handling.
  • Stronger models also put more effort into execution than design, which the authors do not count as a failure.

Checks used

  • Ground truth from human-verified replicable and non-replicable outcomes, with listed data-source URLs for the retrieval task.
  • LLM-based scoring of stages (LLMEval), with 95% confidence intervals across runs; failures counted by category from run logs.

Limits of this case

Only 19 studies, all drawing on web-retrievable data; the authors note that primary data collection is out of scope. Read via the main text and tables, not every appendix.

Search the collection

↑ ↓ move · Enter open · Shift+Enter original source · Esc close · Search index for agents

Keyboard shortcuts

Single keys work when you are not typing and no dialog is open. Turn them off if stray key presses move the page. ⌘K / Ctrl+K and Esc keep working.

/
Search the collection
⌘K / Ctrl+K
Open or close search
?
Show these shortcuts
h
Home
l
Lessons
a
Accounts
r
Replications
p
Papers
g
Guides and tools
d
Timeline
b
Bookmark the record you are reading (press again to remove it)
s
Your saved list and notes
← →
Previous and next record on an account, lesson, replication or paper page, and in a quick look
m
Switch between system, light and dark theme
t
Back to the top of the page
Open full page