MixedReplication study

Add a note

LLM workflow run over 4,452 economics replication packages

What was attempted

The authors describe an open-source workflow that gives an LLM a published article's replication package and asks it to do three things: reproduce the original calculations and flag discrepancies from the published findings, run automated sensitivity analyses, improve the calculations with a different implementation or algorithm, and develop extensions that fit the article's assumptions but are not in it. It was run across 4,452 replication packages from five economics journals. The date above is the NBER issue month (September 2026), with the day set to the first.

What happened

  • Discrepancies were flagged in 3,460 articles or their appendices.
  • Computation time was cut by more than a factor of ten in 496 articles, with similar or greater accuracy.
  • Extensions were developed for 923 articles.
  • The NBER page does not report how many packages reproduced cleanly.

Why it failed or succeeded, per the authors

Not stated on the page read; no failure analysis was available in the abstract. The page states that the LLMs were used in the analysis and writing, and that one author worked as a contractor for Anthropic.

Checks used

The abstract does not say whether flagged discrepancies were audited by a human. The working paper itself was not read.

Limits of this case

Abstract-level record only. A flagged discrepancy is not a confirmed error. Without the full paper it is unclear how many flags are real, how many reflect missing materials, and which model was used. Treat the counts as unverified until the paper is read.

Search the collection

↑ ↓ move · Enter open · Shift+Enter original source · Esc close · Search index for agents

Keyboard shortcuts

Single keys work when you are not typing and no dialog is open. Turn them off if stray key presses move the page. ⌘K / Ctrl+K and Esc keep working.

/
Search the collection
⌘K / Ctrl+K
Open or close search
?
Show these shortcuts
h
Home
l
Lessons
a
Accounts
r
Replications
p
Papers
g
Guides and tools
d
Timeline
b
Bookmark the record you are reading (press again to remove it)
s
Your saved list and notes
← →
Previous and next record on an account, lesson, replication or paper page, and in a quick look
m
Switch between system, light and dark theme
t
Back to the top of the page
Open full page