political scienceblog
Tom Pepinsky
More in social sciences and law
- Thomas D. Grantlaw, 20 Nov 2026
- Chris Blattmaneconomics, Oct 2026
- William Baudelaw, 25 Sep 2026
- Checking an R port against Stataeconomics, 21 Jul 2026
- Pedro H. C. Sant'Annaeconomics, 24 Jun 2026
What they did
Pepinsky, previously a critic focused on what AI cannot do, reports starting to use Claude Code after seeing Andrew Hall, Scott Cunningham and Andrew Little, and calls it a game-changer for social scientists. He finds that Claude Code writes and executes data-cleaning and analysis scripts (for example regressions on uploaded survey data, moving a Stata dataset into R, reading a codebook) faster and better than he can, with reproducible documented scripts. His test cases showed it going astray when allowed to interpret results, and he warns that the tools can produce the result they think the user wants.
In their words
"Upload a CSV file containing survey data to Claude Code and ask it to run a regression analysis and plot the quantities of interest, and it works."
"In several test cases, Claude Code seemed to be guessing what I wanted to hear, interpreting basic regression results incorrectly but provocatively."
"use agentic AI for tasks that involve following rules. Do not use agentic AI for tasks that generate answers, arguments, or interpretations."
Lessons they draw
- Use agentic AI for rule-following tasks (cleaning, scripts, visualisation), not for answers, arguments or interpretation (Pepinsky).
- Citation hallucination remains a problem for LLM literature claims, whereas code can be verified by executing it (Pepinsky).
- Risk of p-hacking: the tool may generate what it "thinks" you want, and if execution is handed over the author may not know what was chosen or left out (Pepinsky).
- As coding costs fall, the relative value of reading, interpreting and understanding rises; he reports seeing AI-written submissions with interpretation problems (Pepinsky).
Limits of this account
Early-use impression (about one month with the tool) with test cases described rather than documented; the interpretation claim rests on his stated hunch.