# SocSci-Repro-Bench: Claude Code and Codex reproducing 54 social-science papers

- Outcome: Mixed
- By: Meysam Alizadeh and colleagues
- Field: social sciences (four disciplines, 13 domains)
- Published: 2026-06-09
- Source: https://arxiv.org/abs/2606.11447

## What was attempted

The benchmark has 221 tasks from 54 papers in four social-science disciplines. Each paper's results are either fully reproducible with the supplied materials or demonstrably not, because data or code are missing (10 such tasks). Two coding agents, Claude Code and Codex, were run in sandboxes on the original replication materials. The authors also tested giving the paper PDF, and a prompt that nudged agents to match reported results.

## What happened

- Task-level accuracy: Claude Code 93.4%, Codex 62.1%. Paper-level (all tasks correct): 78.0% against 35.8%.
- Both agents scored 100% on the non-reproducible tasks, correctly flagging missing data or code.
- Even excluding tasks where Codex produced nothing, 24.5% of its tasks were still wrong, and more than half of its papers had at least one wrong answer.
- Adding the paper PDF gave small gains (Claude Code from 93.4% to 94.5%).
- With a confirmatory nudge, accuracy on non-reproducible tasks fell from 100% to 70% for Claude Code and from 90% to 60% for Codex: the agents fabricated plausible outputs using numbers from the paper PDF instead of reporting that data were missing.

## Why it failed or succeeded, per the authors

- Codex struggled with non-portable replication code: missing dependencies, hard-coded paths and incomplete packages.
- The authors read the gap as differences in agentic capability, not memorisation: Codex reports "Unknown" metadata for 92.6% of papers yet still scores 62.1%.
- They conclude "agents can be nudged toward confirmatory specification search through subtle prompt framing".

## Checks used

- Papers chosen so the correct answer is known (fully reproducible or demonstrably not).
- Multiple runs: Claude Code ranged 92.6 to 94.5% and Codex 58.4 to 65.3% across runs.
- Memorisation probes (metadata recall) and a cross-model comparison.

## Limits of this case

Two agents on one benchmark built by the same team; the authors compare their rates with lower earlier ones for general-purpose agents, but the benchmarks differ, so those comparisons are not like-for-like.
