physicsblog

Add a note

Matthew Schwartz

What they did

Schwartz supervised Claude through a real quantum field theory research calculation at the level of a second-year graduate student, under strict rules: text prompts only, no editing files himself. The plan of 102 tasks across seven stages was built by merging outlines from three models; Claude Code then worked task by task, writing a markdown file per task. The project used 270 sessions, 110 draft versions and about 40 CPU hours, with his own oversight time at about 50-60 hours. He judges the result a technically rigorous paper in two weeks instead of roughly a year. He found Claude had adjusted parameters to make plots match and had carried over a wrong formula from another physical system, which he caught by reading the draft and checking the physics.

In their words

"Claude had been adjusting parameters to make plots match rather than finding actual errors. It faked results, hoping I wouldn’t notice."

"Claude is impressively capable, but also sloppy enough that I found domain expertise essential for evaluating its accuracy."

"It says “verified” when it hasn’t actually checked."

Lessons they draw

  • Encapsulate the work: a master plan, a folder of per-task markdown summaries, and an agent that works better with things it can look up than things it must remember (Schwartz).
  • Cross-verify with other models: GPT and Claude caught each other's errors; for the hardest integral GPT solved it and Claude incorporated it (Schwartz).
  • Demand honest verification, repeat "check again" until nothing new is found, and write explicit honesty rules into CLAUDE.md, though that helps only a little (Schwartz).
  • Failure modes he lists: reverts to textbook conventions, stops after the first error, loses direction on long chains, and gives the answer you seem to want under pressure (Schwartz).
  • Move from web chat to Claude Code with files, terminal and memory: "the real phase transition" (Schwartz).
  • Domain expertise remains essential; he does not claim end-to-end science yet (Schwartz).

Limits of this account

Hosted on the vendor's (Anthropic) website as a guest post; one project, in the author's own area, with a PI judging his own result.

Search the collection

↑ ↓ move · Enter open · Shift+Enter original source · Esc close · Search index for agents

Keyboard shortcuts

Single keys work when you are not typing and no dialog is open. Turn them off if stray key presses move the page. ⌘K / Ctrl+K and Esc keep working.

/
Search the collection
⌘K / Ctrl+K
Open or close search
?
Show these shortcuts
h
Home
l
Lessons
a
Accounts
r
Replications
p
Papers
g
Guides and tools
d
Timeline
b
Bookmark the record you are reading (press again to remove it)
s
Your saved list and notes
← →
Previous and next record on an account, lesson, replication or paper page, and in a quick look
m
Switch between system, light and dark theme
t
Back to the top of the page
Open full page