MixedAudit of an AI system
Independent evaluation of Sakana's AI Scientist
Also mixed
- LLM workflow run over 4,452 economics replication packages1 Sep 2026
- Coding agents replicating four scientific machine learning papers with an evidence workflow2 Jul 2026
- SocSci-Repro-Bench: Claude Code and Codex reproducing 54 social-science papers9 Jun 2026
- Institute for Replication AI replication games: human-only, AI-assisted and AI-led teams26 Mar 2026
- Auditing three Kosmos radiation-biology claims against null models17 Nov 2025
- Kosmos reproducing unpublished and preprinted findings (developer-run)4 Nov 2025
What was attempted
The authors installed the open-source AI Scientist and ran it end to end on recommender-systems research ideas, to test Sakana's claims that it automates the full research lifecycle. They supplied two seed ideas, generated further ideas, ran the experiments and read the resulting manuscripts. This is a test of a system's claims rather than of one published paper. The Zotero title is the first version's title; the arXiv v3 read here has a different title.
What happened
- Five of twelve proposed experiments (42%) failed to run because of coding errors; the system flagged 10 generated ideas and both seeds as novel, including well-established ones such as micro-batching for SGD.
- Experiments that ran often gave logically flawed results; in one, an energy-efficiency experiment reported accuracy gains while using more compute.
- Code edits were small, about 8% more characters in the first iteration, shrinking in later ones.
- Seven manuscripts were produced; median five citations each, five of 34 citations from 2020 or later, four of seven manuscripts with missing or misplaced figures or placeholder text, and four of seven with incorrect or hallucinated numbers.
- Cost was roughly $6 per manuscript in API charges.
Why it failed or succeeded, per the authors
- Novelty checking relies on keyword searches over Semantic Scholar, not synthesis of the literature.
- The system depends on the supplied templates and cannot critically assess its own results.
- The authors conclude it is a notable advance in producing complete manuscripts, with quality like that of a rushed undergraduate.
Checks used
- Manual reading of every generated manuscript and experiment log.
- Comparison against Sakana's stated claims.
- A containerised reproduction package published so others can rerun the evaluation.
Limits of this case
Small sample (twelve experiments, seven manuscripts) in one subfield, on one version of the tool. The headline claim, "five out of twelve proposed experiments (42%) failed due to coding errors", comes from that sample.