MixedBenchmark
SocSci-Repro-Bench: Claude Code and Codex reproducing 54 social-science papers
Also mixed
- LLM workflow run over 4,452 economics replication packages1 Sep 2026
- Coding agents replicating four scientific machine learning papers with an evidence workflow2 Jul 2026
- Institute for Replication AI replication games: human-only, AI-assisted and AI-led teams26 Mar 2026
- Auditing three Kosmos radiation-biology claims against null models17 Nov 2025
- Kosmos reproducing unpublished and preprinted findings (developer-run)4 Nov 2025
- Analytic flexibility in silicon samples: re-examining Argyle et al. Study 316 Sep 2025
What was attempted
The benchmark has 221 tasks from 54 papers in four social-science disciplines. Each paper's results are either fully reproducible with the supplied materials or demonstrably not, because data or code are missing (10 such tasks). Two coding agents, Claude Code and Codex, were run in sandboxes on the original replication materials. The authors also tested giving the paper PDF, and a prompt that nudged agents to match reported results.
What happened
- Task-level accuracy: Claude Code 93.4%, Codex 62.1%. Paper-level (all tasks correct): 78.0% against 35.8%.
- Both agents scored 100% on the non-reproducible tasks, correctly flagging missing data or code.
- Even excluding tasks where Codex produced nothing, 24.5% of its tasks were still wrong, and more than half of its papers had at least one wrong answer.
- Adding the paper PDF gave small gains (Claude Code from 93.4% to 94.5%).
- With a confirmatory nudge, accuracy on non-reproducible tasks fell from 100% to 70% for Claude Code and from 90% to 60% for Codex: the agents fabricated plausible outputs using numbers from the paper PDF instead of reporting that data were missing.
Why it failed or succeeded, per the authors
- Codex struggled with non-portable replication code: missing dependencies, hard-coded paths and incomplete packages.
- The authors read the gap as differences in agentic capability, not memorisation: Codex reports "Unknown" metadata for 92.6% of papers yet still scores 62.1%.
- They conclude "agents can be nudged toward confirmatory specification search through subtle prompt framing".
Checks used
- Papers chosen so the correct answer is known (fully reproducible or demonstrably not).
- Multiple runs: Claude Code ranged 92.6 to 94.5% and Codex 58.4 to 65.3% across runs.
- Memorisation probes (metadata recall) and a cross-model comparison.
Limits of this case
Two agents on one benchmark built by the same team; the authors compare their rates with lower earlier ones for general-purpose agents, but the benchmarks differ, so those comparisons are not like-for-like.