MixedReplication study
Coding agents replicating four scientific machine learning papers with an evidence workflow
Also mixed
- LLM workflow run over 4,452 economics replication packages1 Sep 2026
- SocSci-Repro-Bench: Claude Code and Codex reproducing 54 social-science papers9 Jun 2026
- Institute for Replication AI replication games: human-only, AI-assisted and AI-led teams26 Mar 2026
- Auditing three Kosmos radiation-biology claims against null models17 Nov 2025
- Kosmos reproducing unpublished and preprinted findings (developer-run)4 Nov 2025
- Analytic flexibility in silicon samples: re-examining Argyle et al. Study 316 Sep 2025
What was attempted
The authors wrote a workflow, packaged as a coding-agent skill, in which the agent records each computational claim of a paper as a target, rebuilds the method, runs experiments, and links outputs to provenance and to the paper's claim. Completion has to pass external validation checks, not the agent's own statement. They ran three independent replications from paper materials alone, with author code excluded, on each of four papers (PIFT, PINN-I, PINN-II, SINDy), twelve runs in all. At least one of the four papers (PIFT) is co-authored by one of the authors, so this is a mixed case of own and others' work; authorship of the other three was not checked.
What happened
- All twelve workspaces passed the completion gate and all 158 recorded targets were matched with report coverage.
- Of 39 scalar anchor-run observations (13 standardised anchors, three runs each), 37 fell inside the paper-anchored threshold; two did not (a Schrödinger relative L2 error of 4.8 × 10^-2 against a 1% threshold, and a Navier–Stokes coefficient error of 16.4% against 10%).
- Elapsed time ranged from 1.2 to 13.0 hours; posterior median 6.9 hours for PINN-II and 5.0 for PINN-I against 2.2 for PIFT and 1.9 for SINDy.
- Reproduced PINN tables kept the reported accuracy patterns, but individual cell values did not match every printed digit.
- For a cylinder-wake target the paper materials omit the simulation trajectories and preprocessing, so the matched evidence supports the wake structure and trajectory geometry, not exact coefficient equality.
Why it failed or succeeded, per the authors
- The authors say a plain prompt does not reliably preserve progress or check that generated evidence supports claims, which the external gate fixes.
- "repeated runs differ in how papers are divided into targets, in numerical fidelity", and in effort and accepted evidence rules, even when all pass.
- Papers without seeds, sampled training sets or preprocessing details limit exact replication.
Checks used
- Workspace validation gate with hashes and provenance for every matched target.
- A separate paper-anchored scalar threshold analysis, independent of the per-target acceptance rule.
- Three independent runs per paper.
Limits of this case
Four papers and one agent (Codex with GPT-5.4); "matched" means the target-specific acceptance rule was met, which can be looser than the paper-anchored threshold. The authors' own workflow is being evaluated on a small set including at least one paper by an author.