political scienceblog

Add a note

Andy Hall

What they did

Hall took his own 2020 paper on vote-by-mail, handed it to an AI coding assistant and asked it to replicate the findings from scratch and extend the analysis with new data. He commissioned an independent audit: a UCLA PhD student, Graham Straus, updated the paper manually and then went through Claude's code line by line against his own ground truth. Hall reports that Claude's estimates were close to Straus's, that it coded 29 of 30 California counties correctly on treatment timing, and that its collected data correlated above .999 with manual figures. It did so in under an hour for about ten dollars.

In their words

"It made mistakes—missed some races, failed to run calculations it had the inputs for, documented poorly."

"every new empirical paper should ship with proof that an AI agent successfully reproduced its results from the raw data."

Lessons they draw

  • Commission an independent manual audit rather than trusting the agent's own report (Hall).
  • Errors he found: missed races, calculations not run despite available inputs, poor documentation (Hall).
  • He proposes that agent-run replication from raw data could become a routine pre-publication check and that findings could be re-run as "living research" after each new election (Hall).

Limits of this account

Author's own paper, audit commissioned by him; the post is partly a vision piece, and the audit write-up itself was not read.

Search the collection

↑ ↓ move · Enter open · Shift+Enter original source · Esc close · Search index for agents

Keyboard shortcuts

Single keys work when you are not typing and no dialog is open. Turn them off if stray key presses move the page. ⌘K / Ctrl+K and Esc keep working.

/
Search the collection
⌘K / Ctrl+K
Open or close search
?
Show these shortcuts
h
Home
l
Lessons
a
Accounts
r
Replications
p
Papers
g
Guides and tools
d
Timeline
b
Bookmark the record you are reading (press again to remove it)
s
Your saved list and notes
← →
Previous and next record on an account, lesson, replication or paper page, and in a quick look
m
Switch between system, light and dark theme
t
Back to the top of the page
Open full page