# Kosmos reproducing unpublished and preprinted findings (developer-run)

- Outcome: Mixed
- By: Ludovico Mitchener and colleagues
- Field: metabolomics, materials science, neuroscience, statistical genetics
- Published: 2025-11-04
- Source: https://arxiv.org/abs/2511.02824

## What was attempted

The developers gave Kosmos open-ended objectives and datasets, then checked whether its reports matched findings that were unpublished, preprinted or published after the underlying models' cut-off. Three of the seven highlighted discoveries are framed this way: a metabolomics dataset on cooling-induced neuroprotection (later preprinted), a perovskite solar-cell fabrication dataset, and neuron-morphology data from eight connectomes (matching a bioRxiv preprint). Separately, expert scientists audited 102 statements from three reports for accuracy.

## What happened

- Metabolomics run: nine of the top fifteen metabolites matched the original analysis exactly, with log10 fold changes closely matching (R2 = 0.998) and the same nucleotide-salvage conclusion.
- Connectome run: Kosmos reproduced the two main quantitative results of the preprint (lognormal distributions; power-law relations between metrics).
- Statement audit: 79.4% of statements accurate overall; 85.5% of data-analysis statements reproducible, 82.1% of literature statements validated, and 57.9% of interpretive synthesis statements accurate.
- The seven highlighted discoveries include four novel ones that were not reproductions.

## Why it failed or succeeded, per the authors

- Success is attributed to a shared world model linking data-analysis and literature agents over about 200 rollouts.
- Weakness is in interpretation: Kosmos tends to conflate statistical significance with scientific value, and invents unorthodox metrics.
- In the metabolomics run its extra claim about de novo synthesis could not be settled, because the data were steady-state levels only.

## Checks used

- Expert scientists classified sampled statements as supported or refuted, without being told which came from which analysis type.
- Human-validation plots compared against Kosmos output, and the original studies as reference.
- "Independent scientists found 79.4% of statements in Kosmos reports to be accurate", per the abstract.

## Limits of this case

Developer-run and selected: the seven highlighted cases were picked from collaborations, and failed or uninteresting runs are not itemised. See the independent audit of Kosmos in `kosmos-radiation-biology-audit.md` for a harsher picture.
