# PaperBench: agents replicating 20 ICML 2024 papers from scratch

- Outcome: Partly succeeded
- By: Giulio Starace and colleagues
- Field: machine learning
- Published: 2025-04-02
- Source: https://arxiv.org/abs/2504.01848

## What was attempted

Agents were given the text of 20 ICML 2024 Spotlight and Oral papers plus an addendum of clarifications. They had to write a codebase from scratch, run it, and produce a `reproduce.sh` entry point that regenerates the paper's empirical results. Each paper has a hierarchical rubric (8,316 gradable outcomes in total) that was co-developed with one of the original authors. Submissions were re-run and graded by an LLM judge against the rubric. The agents were barred from using the authors' own code repositories, which were blacklisted.

## What happened

- Best agent on the main benchmark: Claude 3.5 Sonnet (New) with a basic scaffold, average replication score 21.0%. OpenAI o1 scored 13.2%; the other models scored under 10%.
- A variant scaffold that stops agents ending early (IterativeAgent) lifted o1-high to 24.4% and o3-mini-high to 8.5%, but lowered Claude 3.5 Sonnet to 16.1%. A 36-hour o1 run reached 26.0%.
- Code-Dev variant (code written but not executed or graded on results): o1 scored 43.4%.
- Human baseline: eight ML PhD students attempted a subset. The introduction says a 3-paper subset, section 5.4 says 4 papers; best-of-3 humans scored 41.4% after 48 hours against 26.6% for o1 on the same subset. o1 led in the first hours and humans overtook it after about 24 hours.

## Why it failed or succeeded, per the authors

- Models other than Claude 3.5 Sonnet often stopped early, claiming they had finished or hit an unsolvable problem; "frequently finished early, claiming that they either had finished the entire replication or had faced a problem they couldn’t solve".
- All agents failed to plan how to use the limited time; o3-mini often struggled with tool use.
- The authors read this as weak long-horizon behaviour: good at writing plans, poor at carrying them out.
- Scaffold prompts that helped o-series models hurt Claude, so scores depend on scaffold as well as model.

## Checks used

- Rubrics co-developed with original paper authors.
- LLM judge validated on a separate gold-label set graded by human experts (best judge F1 0.83).
- Three runs per paper; blacklist check for use of forbidden resources across 646 runs.
- Human expert baseline under similar conditions.

## Limits of this case

Replicates ML code and experiments, where compute and code are the only barriers; the LLM judge is a stand-in for human grading. Headline numbers are from the first version of the paper held in Zotero.
