MixedReplication study
Kosmos reproducing unpublished and preprinted findings (developer-run)
Also mixed
- LLM workflow run over 4,452 economics replication packages1 Sep 2026
- Coding agents replicating four scientific machine learning papers with an evidence workflow2 Jul 2026
- SocSci-Repro-Bench: Claude Code and Codex reproducing 54 social-science papers9 Jun 2026
- Institute for Replication AI replication games: human-only, AI-assisted and AI-led teams26 Mar 2026
- Auditing three Kosmos radiation-biology claims against null models17 Nov 2025
- Analytic flexibility in silicon samples: re-examining Argyle et al. Study 316 Sep 2025
What was attempted
The developers gave Kosmos open-ended objectives and datasets, then checked whether its reports matched findings that were unpublished, preprinted or published after the underlying models' cut-off. Three of the seven highlighted discoveries are framed this way: a metabolomics dataset on cooling-induced neuroprotection (later preprinted), a perovskite solar-cell fabrication dataset, and neuron-morphology data from eight connectomes (matching a bioRxiv preprint). Separately, expert scientists audited 102 statements from three reports for accuracy.
What happened
- Metabolomics run: nine of the top fifteen metabolites matched the original analysis exactly, with log10 fold changes closely matching (R2 = 0.998) and the same nucleotide-salvage conclusion.
- Connectome run: Kosmos reproduced the two main quantitative results of the preprint (lognormal distributions; power-law relations between metrics).
- Statement audit: 79.4% of statements accurate overall; 85.5% of data-analysis statements reproducible, 82.1% of literature statements validated, and 57.9% of interpretive synthesis statements accurate.
- The seven highlighted discoveries include four novel ones that were not reproductions.
Why it failed or succeeded, per the authors
- Success is attributed to a shared world model linking data-analysis and literature agents over about 200 rollouts.
- Weakness is in interpretation: Kosmos tends to conflate statistical significance with scientific value, and invents unorthodox metrics.
- In the metabolomics run its extra claim about de novo synthesis could not be settled, because the data were steady-state levels only.
Checks used
- Expert scientists classified sampled statements as supported or refuted, without being told which came from which analysis type.
- Human-validation plots compared against Kosmos output, and the original studies as reference.
- "Independent scientists found 79.4% of statements in Kosmos reports to be accurate", per the abstract.
Limits of this case
Developer-run and selected: the seven highlighted cases were picked from collaborations, and failed or uninteresting runs are not itemised. See the independent audit of Kosmos in kosmos-radiation-biology-audit.md for a harsher picture.