Partly succeededReplication study
Scott Cunningham: Claude Code replicating a PNAS text-classification paper with GPT-4o-mini
Also partly succeeded
- LLM reanalysis of 180 social and behavioural science studies11 Jun 2026
- Reproducing 48 social-science papers from the methods description alone23 Apr 2026
- ReplicatorBench: agents replicating social and behavioural science claims with new data11 Feb 2026
- GPT-4o agent team reproducing five Alzheimer's disease studies29 May 2025
- PaperBench: agents replicating 20 ICML 2024 papers from scratch2 Apr 2025
What was attempted
Cunningham asked Claude Code to replicate the text-classification part of Card et al.'s PNAS paper on 140 years of immigration rhetoric in congressional and presidential speeches. The original used a fine-tuned RoBERTa classifier. He replaced it with zero-shot labels from gpt-4o-mini through the OpenAI Batch API. Claude Code handled the infrastructure, found and downloaded the original replication data, and ran the pipeline over about 305,000 speech segments. The post is part two of a pair; part one described the setup.
What happened
- The LLM labels agreed with the original classifier's labels in 69% of cases.
- The headline substantive findings held: Democrats and Republicans diverge sharply from the 1970s onward under both classifiers, and the country-of-origin ordering (Italy, China, Mexico) held.
- Direct polarity flips between pro and anti were about 4%; where classifiers disagreed the LLM mostly moved speeches to neutral (33% of pro and 44% of anti speeches reclassified as neutral).
- Run cost $10.99 and about 4.6 hours in total (2.6 hours of batch runtime, against 24 hours he had expected).
- He reports a lapse in his own protocol: a "Referee 2" review was run inside the same session, and a reader caught it.
Why it failed or succeeded, per the author
- Cunningham reads the move toward neutral as the LLM being cautious on ambiguous speeches.
- He argues the 69% figure should be read against modest agreement among human coders (Krippendorff's alpha 0.48).
Checks used
- Transition matrix between original and LLM labels, and comparison of the substantive conclusions across both classifiers.
- Human annotator agreement as a benchmark.
- No separate validation step is described in the excerpt read.
Limits of this case
The Substack page was paywalled after the summary, so the first-hand text read covers only the date, the title, the 69% agreement, the cost and the time; the other figures come from a third-party summary of the post (a wiki, velikov-mihail.github.io/ai-econ-wiki). It replicates a classifier-based result, not a full reproduction of the paper. A blog report, not peer reviewed.