# REPRO-Bench: agents assessing reproducibility of 112 social-science papers

- Outcome: Failed
- By: Chuxuan Hu and colleagues
- Field: economics and political science
- Published: 2025-07-25
- Source: https://arxiv.org/abs/2507.18901

## What was attempted

Agents were given a social-science paper (PDF) and its reproduction package, and asked to rate its reproducibility from 1 (not reproducible) to 4 (fully reproducible). Ground truth was taken from published reproduction reports by social scientists: 112 task instances across Stata, R and Python. The task is to execute the package and check consistency with the paper's claims, not only to rerun the code.

## What happened

- Best existing agent (CORE-Agent with GPT-4o): 21.4% accuracy, below the 25% expected by guessing among four scores.
- The authors' REPRO-Agent, with a four-phase template, a fallback score and few-shot examples of common errors, reached 36.6%, a 71% relative gain.
- The authors report that the three agents did about as well on packages with several data formats (52% average accuracy) as on single-format packages (54%).

## Why it failed or succeeded, per the authors

- Four failure types: (1) wrong comparison of reproduced and reported results; (2) Stata scripts whose errors go to log files, not the terminal, so the agent misreads silence; (3) failed library installation; (4) files not in the execution directory, the most frequent type.
- Agents often concluded data were missing without searching the package.
- Reading whole code files instead of relevant sections made inconsistency detection hard.

## Checks used

- Human reproduction reports as ground truth, annotated by the authors in a consensus process.
- Accuracy measured on the 1–4 score.

## Limits of this case

All agents used one 2024 model (GPT-4o); the later SocSci-Repro-Bench and Kohler et al. results (see `socsci-repro-bench.md`, `kohler-read-paper-write-code.md`) show much higher rates with newer coding agents. The abstract reports an "accuracy of only 21.4%".
