MixedReplication study
LLM workflow run over 4,452 economics replication packages
Also mixed
- Coding agents replicating four scientific machine learning papers with an evidence workflow2 Jul 2026
- SocSci-Repro-Bench: Claude Code and Codex reproducing 54 social-science papers9 Jun 2026
- Institute for Replication AI replication games: human-only, AI-assisted and AI-led teams26 Mar 2026
- Auditing three Kosmos radiation-biology claims against null models17 Nov 2025
- Kosmos reproducing unpublished and preprinted findings (developer-run)4 Nov 2025
- Analytic flexibility in silicon samples: re-examining Argyle et al. Study 316 Sep 2025
What was attempted
The authors describe an open-source workflow that gives an LLM a published article's replication package and asks it to do three things: reproduce the original calculations and flag discrepancies from the published findings, run automated sensitivity analyses, improve the calculations with a different implementation or algorithm, and develop extensions that fit the article's assumptions but are not in it. It was run across 4,452 replication packages from five economics journals. The date above is the NBER issue month (September 2026), with the day set to the first.
What happened
- Discrepancies were flagged in 3,460 articles or their appendices.
- Computation time was cut by more than a factor of ten in 496 articles, with similar or greater accuracy.
- Extensions were developed for 923 articles.
- The NBER page does not report how many packages reproduced cleanly.
Why it failed or succeeded, per the authors
Not stated on the page read; no failure analysis was available in the abstract. The page states that the LLMs were used in the analysis and writing, and that one author worked as a contractor for Anthropic.
Checks used
The abstract does not say whether flagged discrepancies were audited by a human. The working paper itself was not read.
Limits of this case
Abstract-level record only. A flagged discrepancy is not a confirmed error. Without the full paper it is unclear how many flags are real, how many reflect missing materials, and which model was used. Treat the counts as unverified until the paper is read.