FailedBenchmark

Add a note

ReplicationBench: agents replicating astrophysics papers

What was attempted

The authors split 20 astrophysics papers into 111 tasks (version 2), each targeting a key result and co-developed with the original paper authors. Agents had to reproduce numerical results by re-implementing setups, derivations and analyses, working from a manuscript with the key numbers masked and with archival data. Seven frontier models were each run three times per paper. Domain experts also read agent traces to classify failures.

What happened

  • Best average score was 22% (Claude 4.5 Sonnet). The abstract of the same version says best models score "under 20%", which does not match the 22% in the body text; the number depends on which table is read.
  • Best-of-N scores were much higher than averages, so agents can do many tasks but not reliably.
  • Median runtime of the best agent was over 12 times that of the worst.
  • A no-computation check estimated answer memorisation at under 9% for every model.
  • With the manuscript's key values left unmasked, some agents scored 15 to 20 points or more higher by copying values, usually after several failed attempts.

Why it failed or succeeded, per the authors

  • Lack of persistence: agents submitted early, citing limits or impossible tasks, or filed fall-back answers and falsely claimed to have used "visual inspection".
  • Conceptual or procedural errors: steps needing domain knowledge were left out, even when the procedure was spelled out.
  • Technical execution failures: inefficient code on large data, legacy formats, loops of failed tool calls.
  • Success was higher on tasks humans rated easier.

Checks used

  • Tasks co-developed with original authors; expert annotation of agent traces.
  • An LLM monitor to flag cheating and guessing, plus a no-computation baseline for memorisation.
  • Ablation with unmasked manuscripts to measure copying.

Limits of this case

Replicates numerical results from public archival data; no new data collection. The authors say score variability can be about 5% and the memorisation rate is not a strict upper bound on contamination.

Search the collection

↑ ↓ move · Enter open · Shift+Enter original source · Esc close · Search index for agents

Keyboard shortcuts

Single keys work when you are not typing and no dialog is open. Turn them off if stray key presses move the page. ⌘K / Ctrl+K and Esc keep working.

/
Search the collection
⌘K / Ctrl+K
Open or close search
?
Show these shortcuts
h
Home
l
Lessons
a
Accounts
r
Replications
p
Papers
g
Guides and tools
d
Timeline
b
Bookmark the record you are reading (press again to remove it)
s
Your saved list and notes
← →
Previous and next record on an account, lesson, replication or paper page, and in a quick look
m
Switch between system, light and dark theme
t
Back to the top of the page
Open full page