Partly succeededBenchmark
PaperBench: agents replicating 20 ICML 2024 papers from scratch
Also partly succeeded
- LLM reanalysis of 180 social and behavioural science studies11 Jun 2026
- Reproducing 48 social-science papers from the methods description alone23 Apr 2026
- ReplicatorBench: agents replicating social and behavioural science claims with new data11 Feb 2026
- Scott Cunningham: Claude Code replicating a PNAS text-classification paper with GPT-4o-mini6 Feb 2026
- GPT-4o agent team reproducing five Alzheimer's disease studies29 May 2025
What was attempted
Agents were given the text of 20 ICML 2024 Spotlight and Oral papers plus an addendum of clarifications. They had to write a codebase from scratch, run it, and produce a reproduce.sh entry point that regenerates the paper's empirical results. Each paper has a hierarchical rubric (8,316 gradable outcomes in total) that was co-developed with one of the original authors. Submissions were re-run and graded by an LLM judge against the rubric. The agents were barred from using the authors' own code repositories, which were blacklisted.
What happened
- Best agent on the main benchmark: Claude 3.5 Sonnet (New) with a basic scaffold, average replication score 21.0%. OpenAI o1 scored 13.2%; the other models scored under 10%.
- A variant scaffold that stops agents ending early (IterativeAgent) lifted o1-high to 24.4% and o3-mini-high to 8.5%, but lowered Claude 3.5 Sonnet to 16.1%. A 36-hour o1 run reached 26.0%.
- Code-Dev variant (code written but not executed or graded on results): o1 scored 43.4%.
- Human baseline: eight ML PhD students attempted a subset. The introduction says a 3-paper subset, section 5.4 says 4 papers; best-of-3 humans scored 41.4% after 48 hours against 26.6% for o1 on the same subset. o1 led in the first hours and humans overtook it after about 24 hours.
Why it failed or succeeded, per the authors
- Models other than Claude 3.5 Sonnet often stopped early, claiming they had finished or hit an unsolvable problem; "frequently finished early, claiming that they either had finished the entire replication or had faced a problem they couldn’t solve".
- All agents failed to plan how to use the limited time; o3-mini often struggled with tool use.
- The authors read this as weak long-horizon behaviour: good at writing plans, poor at carrying them out.
- Scaffold prompts that helped o-series models hurt Claude, so scores depend on scaffold as well as model.
Checks used
- Rubrics co-developed with original paper authors.
- LLM judge validated on a separate gold-label set graded by human experts (best judge F1 0.83).
- Three runs per paper; blacklist check for use of forbidden resources across 646 runs.
- Human expert baseline under similar conditions.
Limits of this case
Replicates ML code and experiments, where compute and code are the only barriers; the LLM judge is a stand-in for human grading. Headline numbers are from the first version of the paper held in Zotero.