computer sciencepaper

Add a note

Max Zimmer, Nico Pelleriti, Christophe Roux, Sebastian Pokutta

What they did

The authors present a framework of methodological rules, written as agent prompts, that turns CLI coding agents into research assistants running inside a sandboxed container. Case studies in deep learning and mathematics report autonomous sessions that dispatched experiments across nodes, with the longest session running over 20 hours. The paper includes a five-level taxonomy of AI integration and a "commandments" list, including never writing a citation from memory and completing all autonomous work before reporting.

In their words

"In practice, our longest autonomous session ran for over 20 hours, dispatching independent experiments across multiple nodes without human intervention."

"Natural-language proofs remain difficult to verify and require manual inspection."

"researchers must invest substantial effort in verifying agent outputs"

Lessons they draw

  • Failure mode found: the agent kept stopping to ask whether to continue, so a rule requires finishing all autonomous work before reporting (Zimmer et al.).
  • One-variable-at-a-time testing was critical; testing normalisation and weight decay separately showed independent, nearly additive gains (Zimmer et al.).
  • Compute the oracle baseline early; in one case it showed 92 percent of the optimum was already achieved, which prevented wasted effort (Zimmer et al.).
  • Audit formulas through numeric tests on small matrices before benchmarking; negative results (9 of 24 experiments) were more informative than positive ones (Zimmer et al.).
  • Never write a citation from memory; agents hallucinate references, and verification is the researcher's responsibility (Zimmer et al.).

Limits of this account

A methods-and-framework paper by authors who built the tool; case studies are the authors' own, and I read the abstract, framework and case-study lessons but not the full related-work section.

Search the collection

↑ ↓ move · Enter open · Shift+Enter original source · Esc close · Search index for agents

Keyboard shortcuts

Single keys work when you are not typing and no dialog is open. Turn them off if stray key presses move the page. ⌘K / Ctrl+K and Esc keep working.

/
Search the collection
⌘K / Ctrl+K
Open or close search
?
Show these shortcuts
h
Home
l
Lessons
a
Accounts
r
Replications
p
Papers
g
Guides and tools
d
Timeline
b
Bookmark the record you are reading (press again to remove it)
s
Your saved list and notes
← →
Previous and next record on an account, lesson, replication or paper page, and in a quick look
m
Switch between system, light and dark theme
t
Back to the top of the page
Open full page