# CORE-Bench: computational reproducibility from code and data, and its later 'solved' status

- Outcome: Mixed
- By: Zachary S. Siegel and colleagues
- Field: computer science, social science, medicine
- Published: 2024-09-17
- Source: https://arxiv.org/abs/2409.11363

## What was attempted

CORE-Bench has 270 tasks drawn from 90 papers with public code, in Python and R. The agent must install dependencies, run the code, and answer questions about the outputs. There are three difficulty levels: results given (Easy), a run command given (Medium), or only a README (Hard). A task counts as solved if every answer falls within a prediction interval from three manual reproductions. The authors tested AutoGPT and their own CORE-Agent with GPT-4o and GPT-4o-mini.

## What happened

- Best agent (CORE-Agent with GPT-4o): 60.0% Easy, 57.8% Medium, 21.5% Hard.
- Vision questions about figures were harder than written ones (59.3% against 87.9% on Easy, for GPT-4o).
- Later update, from the Princeton HAL team (3 December 2025): Claude Code with Claude Opus 4.5 scored 77.8% on Hard under automatic grading and 95.5% after manual validation; the CORE-Agent scaffold with the same model scored about 42%. They declared the benchmark solved and fixed grading errors in 8 tasks, and removed one task with a dead dataset URL.

## Why it failed or succeeded, per the authors

- Easy failures: picking the wrong figure or file when outputs were spread across several.
- Hard failures: agents did not finish resolving dependency versions before hitting the cost limit and kept retrying the same install; raising the cost limit did not help.
- Of the 2025 update: the automatic grader penalised tiny floating-point differences, underspecified tasks and correct-but-different methods; the authors say scaffold choice alone moved Opus 4.5 from 42% to 78%.

## Checks used

- Three manual reproductions per task to define the prediction interval.
- Three repeated agent runs with confidence intervals.
- For the 2025 update, manual re-grading of disagreements.

## Limits of this case

The 2025 update comes from a leaderboard note and a social post by the HAL team, read via their pages (the post's full text was only partly visible), not from a paper. The benchmark checks that code reruns and outputs are read correctly, not whether the paper's claims are right.
