# AI-assisted full-paper replication of 384 political science studies

- Outcome: Succeeded
- By: Yiqing Xu and colleagues
- Field: political science
- Published: 2026-02-17
- Source: https://arxiv.org/abs/2602.16733

## What was attempted

The authors built a workflow that retrieves a paper's replication materials, rebuilds the computing environment, runs the code, and matches outputs to point estimates in the regression tables. They defined all empirical papers in three top political science journals from 2010 to 2025 and drew a stratified sample of 384 studies (8 per journal-year), covering 3,523 models. A second application re-ran 84 instrumental-variable studies and extracted 597 IV specifications for diagnostics. The numbers below are from version 3 of the preprint (31 May 2026); earlier versions reported different figures.

## What happened

- Fully or largely reproducible: 82.5% of papers after DA-RT adoption against 20.8% before; given an accessible replication package, 92.1% (234 of 254).
- Of the 254 with accessible materials, 193 (76.0%) were fully reproducible, meaning every reported coefficient matched exactly, and 41 largely reproducible (more than 80% of coefficients matched).
- 191 papers were not fully reproducible; 130 of them lacked a package or had restricted data or code.
- Among the 61 papers with accessible materials below the strict threshold, 10 were partially and 10 not reproducible.
- IV corpus: all main-text regressions replicated in each of 84 papers, 1,910 models.

## Why it failed or succeeded, per the authors

- "Failures trace almost entirely to missing or restricted data, not to code quality."
- Repeated issues in package structure were encoded as rules and version-controlled across runs, so coverage improved over versions.
- Journal verification requirements plus data-archiving mandates drive reproducibility, not the AI tool.

## Checks used

- Coefficient-level matching against the published tables, with four verdict categories.
- Human review of failures; benchmark verification against a prior IV corpus (Lal et al. 2024).
- One IV paper dropped because table extraction produced no coefficients.

## Limits of this case

It re-executes the authors' own code on their data, which is easier than re-implementing; it cannot rescue missing packages. Strong-sharing journals only. Read the abstract, introduction, results and supplementary tables, not every appendix.
