political scienceblog
Andy Hall
More in social sciences and law
- Thomas D. Grantlaw, 20 Nov 2026
- Chris Blattmaneconomics, Oct 2026
- William Baudelaw, 25 Sep 2026
- Checking an R port against Stataeconomics, 21 Jul 2026
- Pedro H. C. Sant'Annaeconomics, 24 Jun 2026
What they did
Hall took his own 2020 paper on vote-by-mail, handed it to an AI coding assistant and asked it to replicate the findings from scratch and extend the analysis with new data. He commissioned an independent audit: a UCLA PhD student, Graham Straus, updated the paper manually and then went through Claude's code line by line against his own ground truth. Hall reports that Claude's estimates were close to Straus's, that it coded 29 of 30 California counties correctly on treatment timing, and that its collected data correlated above .999 with manual figures. It did so in under an hour for about ten dollars.
In their words
"It made mistakes—missed some races, failed to run calculations it had the inputs for, documented poorly."
"every new empirical paper should ship with proof that an AI agent successfully reproduced its results from the raw data."
Lessons they draw
- Commission an independent manual audit rather than trusting the agent's own report (Hall).
- Errors he found: missed races, calculations not run despite available inputs, poor documentation (Hall).
- He proposes that agent-run replication from raw data could become a routine pre-publication check and that findings could be re-run as "living research" after each new election (Hall).
Limits of this account
Author's own paper, audit commissioned by him; the post is partly a vision piece, and the audit write-up itself was not read.