Partly succeededBenchmark

Add a note

PaperBench: agents replicating 20 ICML 2024 papers from scratch

What was attempted

Agents were given the text of 20 ICML 2024 Spotlight and Oral papers plus an addendum of clarifications. They had to write a codebase from scratch, run it, and produce a reproduce.sh entry point that regenerates the paper's empirical results. Each paper has a hierarchical rubric (8,316 gradable outcomes in total) that was co-developed with one of the original authors. Submissions were re-run and graded by an LLM judge against the rubric. The agents were barred from using the authors' own code repositories, which were blacklisted.

What happened

  • Best agent on the main benchmark: Claude 3.5 Sonnet (New) with a basic scaffold, average replication score 21.0%. OpenAI o1 scored 13.2%; the other models scored under 10%.
  • A variant scaffold that stops agents ending early (IterativeAgent) lifted o1-high to 24.4% and o3-mini-high to 8.5%, but lowered Claude 3.5 Sonnet to 16.1%. A 36-hour o1 run reached 26.0%.
  • Code-Dev variant (code written but not executed or graded on results): o1 scored 43.4%.
  • Human baseline: eight ML PhD students attempted a subset. The introduction says a 3-paper subset, section 5.4 says 4 papers; best-of-3 humans scored 41.4% after 48 hours against 26.6% for o1 on the same subset. o1 led in the first hours and humans overtook it after about 24 hours.

Why it failed or succeeded, per the authors

  • Models other than Claude 3.5 Sonnet often stopped early, claiming they had finished or hit an unsolvable problem; "frequently finished early, claiming that they either had finished the entire replication or had faced a problem they couldn’t solve".
  • All agents failed to plan how to use the limited time; o3-mini often struggled with tool use.
  • The authors read this as weak long-horizon behaviour: good at writing plans, poor at carrying them out.
  • Scaffold prompts that helped o-series models hurt Claude, so scores depend on scaffold as well as model.

Checks used

  • Rubrics co-developed with original paper authors.
  • LLM judge validated on a separate gold-label set graded by human experts (best judge F1 0.83).
  • Three runs per paper; blacklist check for use of forbidden resources across 646 runs.
  • Human expert baseline under similar conditions.

Limits of this case

Replicates ML code and experiments, where compute and code are the only barriers; the LLM judge is a stand-in for human grading. Headline numbers are from the first version of the paper held in Zotero.

Search the collection

↑ ↓ move · Enter open · Shift+Enter original source · Esc close · Search index for agents

Keyboard shortcuts

Single keys work when you are not typing and no dialog is open. Turn them off if stray key presses move the page. ⌘K / Ctrl+K and Esc keep working.

/
Search the collection
⌘K / Ctrl+K
Open or close search
?
Show these shortcuts
h
Home
l
Lessons
a
Accounts
r
Replications
p
Papers
g
Guides and tools
d
Timeline
b
Bookmark the record you are reading (press again to remove it)
s
Your saved list and notes
← →
Previous and next record on an account, lesson, replication or paper page, and in a quick look
m
Switch between system, light and dark theme
t
Back to the top of the page
Open full page