# ReplicatorBench: agents replicating social and behavioural science claims with new data

- Outcome: Partly succeeded
- By: Bang Nguyen and colleagues
- Field: social and behavioural science
- Published: 2026-02-11
- Source: https://arxiv.org/abs/2602.11354

## What was attempted

Unlike code-rerun benchmarks, this one asks agents to replicate a claim: find and retrieve new data, design the analysis, run it, and judge whether the claim replicates. It has 19 observational studies from the Center for Open Science's SCORE project, including human-verified non-replicable ones. Agents are scored on about 1,568 rubric checkpoints across extraction and data retrieval, design and execution, and interpretation. The authors tested their ReplicatorAgent with four LLMs and varied the programming language and level of code access.

## What happened

- Judging replicable versus non-replicable (version 3 numbers): GPT-5 accuracy 76.9% (macro F1 76.9), GPT-4o 61.5%, GPT-5-mini 48.7% (macro F1 47.9).
- Data retrieval was the weak stage: macro F1 about 8 to 19 across settings, and the agent found all of a study's data sources (hit@all) in only 5 to 16% of studies.
- GPT-5 and o3 had far fewer input-data errors than GPT-4o in execution (6 against 211 for GPT-5 and GPT-4o).

## Why it failed or succeeded, per the authors

- Agents "struggle with retrieving new data, necessary to replicate a claim", while designing and running analyses went well.
- GPT-5-mini failed more on syntax and library use than on data handling.
- Stronger models also put more effort into execution than design, which the authors do not count as a failure.

## Checks used

- Ground truth from human-verified replicable and non-replicable outcomes, with listed data-source URLs for the retrieval task.
- LLM-based scoring of stages (LLMEval), with 95% confidence intervals across runs; failures counted by category from run logs.

## Limits of this case

Only 19 studies, all drawing on web-retrievable data; the authors note that primary data collection is out of scope. Read via the main text and tables, not every appendix.
