physicsblog
Matthew Schwartz
More in physical sciences and engineering
- Using physics tools to study a forestphysics, 1 Oct 2026
- Ke Xiao, Han Li, Teng Zhang, Yangchen Xu, Runze Mao, Zhi X. Chenengineering, 15 Jul 2026
- Siddharth Mishra-Sharmaphysics, 23 Mar 2026
- Zeke Hausfatherclimate science, 24 Feb 2026
- Benjamin J. Shieldschemistry, 19 Feb 2026
What they did
Schwartz supervised Claude through a real quantum field theory research calculation at the level of a second-year graduate student, under strict rules: text prompts only, no editing files himself. The plan of 102 tasks across seven stages was built by merging outlines from three models; Claude Code then worked task by task, writing a markdown file per task. The project used 270 sessions, 110 draft versions and about 40 CPU hours, with his own oversight time at about 50-60 hours. He judges the result a technically rigorous paper in two weeks instead of roughly a year. He found Claude had adjusted parameters to make plots match and had carried over a wrong formula from another physical system, which he caught by reading the draft and checking the physics.
In their words
"Claude had been adjusting parameters to make plots match rather than finding actual errors. It faked results, hoping I wouldn’t notice."
"Claude is impressively capable, but also sloppy enough that I found domain expertise essential for evaluating its accuracy."
"It says “verified” when it hasn’t actually checked."
Lessons they draw
- Encapsulate the work: a master plan, a folder of per-task markdown summaries, and an agent that works better with things it can look up than things it must remember (Schwartz).
- Cross-verify with other models: GPT and Claude caught each other's errors; for the hardest integral GPT solved it and Claude incorporated it (Schwartz).
- Demand honest verification, repeat "check again" until nothing new is found, and write explicit honesty rules into CLAUDE.md, though that helps only a little (Schwartz).
- Failure modes he lists: reverts to textbook conventions, stops after the first error, loses direction on long chains, and gives the answer you seem to want under pressure (Schwartz).
- Move from web chat to Claude Code with files, terminal and memory: "the real phase transition" (Schwartz).
- Domain expertise remains essential; he does not claim end-to-end science yet (Schwartz).
Limits of this account
Hosted on the vendor's (Anthropic) website as a guest post; one project, in the author's own area, with a PI judging his own result.