Most of what gets said about AI agents in research is either hype or dismissal. This page collects what researchers who use agents such as Claude Code or Codex say about it themselves. They describe what they handed over, what went wrong and how they checked, and every account links to its source.
The page has 38 accounts from researchers across many disciplines, with the most recent accounts first and accounts dated only to a year at the end. Accounts written before March 2026 are flagged with the models current at the time, because models and agent tools have changed fast. The seven lessons draw first on accounts from June 2026 onwards and use earlier examples where they explain a specific point. Earlier accounts, including those from 2025, are still in the collection. There are 20 records of systematic attempts to have agents or language models replicate published research, also with the most recent first. And there are notes on five key papers: Paper2Agent, the agentic AI methods primer, and three 2026 Nature papers on AI scientist systems.
The first three lessons ask what makes a question worth answering, how to build the method and its checks together, and where a researcher's judgement changes the work. The thread running through the lessons is that agents are most dependable where the output can be checked, and the work is in building those checks. Where there is a right answer, that means a check target the agent can test itself against. Where there is not, it means cheap 'checksums': a second, independent route to the same result, or a way of making the result visible so that someone who knows the field can judge it. The replication studies point the same way: agents do well rerunning a package of code and data that the authors supplied, and much worse rebuilding the analysis from the paper alone. And a score a system gives its own work is not a check. Much of what matters is unglamorous: keeping the plan in a file, using version control, comparing counts against an independent source.
Take these accounts as reported practice, mostly from enthusiasts. People write up what worked, so they are not evidence of typical results. The collection was assembled and summarised with AI assistance from public sources. It is a working collection, not a survey.
Nature Reviews Methods Primers 6:81
Agentic AI systems for scientific discovery
Feng et al. · doi:10.1038/s43586-026-00534-y
A methods primer from a large chemistry and materials group (Aspuru-Guzik, Schwaller, Zheng and others). It treats agentic AI as a research method with its own standards for design, validation and reporting, in the way a primer would treat mass spectrometry. Its examples lean towards chemistry, materials and self-driving labs, but the reporting and validation sections apply to any field.
Full reading notes
Agentic AI systems for scientific discovery (Feng et al., Nat Rev Methods Primers, 2026)
What the paper is
A methods primer from a large chemistry and materials group (Aspuru-Guzik, Schwaller, Zheng and others). It treats agentic AI as a research method with its own standards for design, validation and reporting, in the way a primer would treat mass spectrometry. Its examples lean towards chemistry, materials and self-driving labs, but the reporting and validation sections apply to any field.
The framework in brief
- An agentic system is one or more agents, the workflow that organises them, and the environment they act in (data, software, instruments, people). Agency is a continuum, not a yes/no property: how much discretion over actions, how much effect on the world, how long the task runs, how persistent the goals and state are.
- Three families: cognitive agents (literature, reasoning, computation), embodied agents (instruments and robots) and collaborative agents (coordination among agents and people).
- Building blocks: a profile (often a versioned Markdown file, as in Claude Code), perception, tools (often via MCP), and memory (Markdown files for small structured knowledge; vector stores or graphs for large corpora). They explicitly name Claude Code, Codex, Gemini CLI, OpenCode and reusable skills folders as the practical form.
- The worked example (lipid nanoparticle formulation) is essentially a repository workspace with skills, data files and a results spreadsheet. Human approval is reserved for irreversible, safety-critical or physical actions.
The parts most useful for training
Staged validation (Table 2). Check outputs in order: internal (is it coherent, stable across runs, and is every claim traceable to a source or tool output?), then computational (benchmarks, simulations), then experimental. Cheap internal checks catch most obvious failures.
Separate utility from evidence. An output can usefully steer the next step long before it is evidence for a conclusion. Ungrounded outputs may be kept as hypotheses but must be labelled as unsupported.
Report the degree of autonomy honestly. Say who set the goal, chose the tools, fixed failures, approved irreversible actions and picked which outputs to validate. Most reported "autonomous" systems are a mix of model capability, design and human scaffolding.
Run it more than once. Agents are stochastic and path-dependent; showing only the best run overstates capability. The primer distinguishes technical replicates (same configured system, same prompt, again) from independent replicates (rebuild the system from its documentation and rerun everything).
Reproducibility record. Code and data are not enough. Record prompts, tool versions and interfaces, memory and retrieval set-up, intermediate steps, model versions with access dates, and every human intervention. Treat tools as lab instruments and document them as such. Undocumented manual repair, post hoc selection of candidates and silent changes to the search space are named as threats to reproducibility.
Named failure modes, each with a test or mitigation:
- Error accumulation over long chains; mitigated by separating planning from execution and replanning.
- Mirage reasoning: fluent conclusions from evidence that was never supplied. Test it by removing or altering the evidence and checking whether the conclusion changes.
- Inherited bias from training data; limited by randomisation and blinding of which conditions get tested.
- Context saturation: in long runs the original protocol is silently summarised away and the agent quietly breaks a constraint set at the start.
- Tool misuse: wrong tool, wrong parameters, wrong stage.
- Multi-agent coordination: adding agents can make things worse.
- Prompt injection through material the agent reads (web pages, supplementary files, database fields).
Accountability. An agent cannot be held responsible; named researchers sign off. Also flagged: energy cost (use smaller models for routine steps) and inequity of access.
Limits for our purposes
Written from the perspective of experimental natural science and lab automation. It says little about the everyday case of one researcher with a coding agent and a laptop, which is most of Dominik's audience, and almost nothing about qualitative or humanities research. The principles transfer; the examples need replacing.
Nature 658, 1272–1279
Reimagining research papers as interactive and reliable AI agents
Miao et al. · doi:10.1038/s41586-026-11044-y
The Stanford group (Zou and Pritchard labs) built Paper2Agent, a pipeline that turns a published computational paper into an agent you can talk to. It reads the paper and its code repository and packages the method as an MCP server with three parts: tools (the paper's methods as callable functions), resources (manuscript, supplements, data links) and prompts (the paper's multi-step workflows written as instructions). The researcher then connects that server to a general agent, in their case Claude Code running Claude Sonnet 4, and asks for analyses in ordinary language.
Full reading notes
Paper2Agent (Miao et al., Nature, 2026)
What the paper does
The Stanford group (Zou and Pritchard labs) built Paper2Agent, a pipeline that turns a published computational paper into an agent you can talk to. It reads the paper and its code repository and packages the method as an MCP server with three parts: tools (the paper's methods as callable functions), resources (manuscript, supplements, data links) and prompts (the paper's multi-step workflows written as instructions). The researcher then connects that server to a general agent, in their case Claude Code running Claude Sonnet 4, and asks for analyses in ordinary language.
The pipeline itself is built from Claude Code agents: one sets up the software environment, one extracts the methods into tools, and a testing agent runs each tool against the paper's own reference outputs until the numbers and figures match. Tools that keep failing are dropped. Each surviving tool carries a link back to the original source code.
Results worth quoting
- The AlphaGenome agent took about 45 minutes and US $14 to build on a laptop, with 22 tools, all passing validation.
- On benchmark questions it scored 98.7% (tutorial-style) and 100% (novel queries), against 82.7% and 78.7% for Claude Code given the raw repository, and 37.3% and 56.0% for Biomni. On 30 open-ended researcher-style questions the margin was 82.7% against 56.7%.
- Across 100 computational biology papers, 74 could be agentified. The failures came from missing code, missing data or model files, broken environments and scripts that only worked for one case.
- The AlphaGenome agent re-examined a published LDL-cholesterol result and favoured SORT1 where the original paper emphasised CELSR2 and PSRC1. The authors present this as the agent re-opening a published interpretation, and they note that all three genes have strong evidence.
- In the psoriasis case, three paper agents (AlphaGenome, an MPRA-coupled CRISPRi screen, a Perturb-seq dataset) were combined. The agent proposed ten validation strategies; the human researcher chose one, and that analysis supported GPR137. The authors say the cross-dataset correlation method was not in any of the source papers.
The authors' own caveats
- "Open-ended scientific reasoning, including hypothesis generation and mechanistic interpretation, remains human-in-the-loop." Researchers select directions and evaluate evidence.
- Benchmarks that check agreement with one reference answer measure faithful execution, not analytical validity; open-ended questions can have several defensible answers.
- Paper agents need maintenance as dependencies change, and they raise security, intellectual property and attribution questions.
Ideas that travel beyond genomics
- "How easily can this paper be turned into an agent?" is a reproducibility test. A quarter of repositories failed for ordinary reasons: incomplete code and undocumented environments.
- An "agent availability" statement alongside data and code availability, which the authors predict journals will start to ask for.
- Validate the agent's tools against known answers before trusting it with new questions. The gain over "Claude Code plus the repo" comes almost entirely from this locked, tested layer.
- Encoded workflows beat clever prompting. The MCP prompts carry the correct order of steps, so the user does not have to know it.
- The division of labour in the discovery case: the agent generates options and executes; the human picks.
Limits for our purposes
The case studies are computational biology with well-maintained Python code. The ten non-biology papers (economics, statistics, astrophysics, game theory) show the pipeline runs elsewhere, but the paper says little about qualitative, archival or humanities research, where "the method" is not a function. All experiments used one model family (Claude). The headline accuracy numbers come from benchmarks the authors constructed.
Nature 655, 497–505 (page range taken from the running footers)
A multi-agent system for automating scientific discovery
Ghareeb et al. · doi:10.1038/s41586-026-10652-y
A FutureHouse and Fordham team describes Robin, a workflow of language agents that, given a disease name, proposes in vitro assays and drug candidates and then analyses the laboratory data a human produces. Two literature agents built on PaperQA2, Crow (short reviews) and Falcon (long reports on each candidate), feed an LLM-judged pairwise tournament that ranks assays and candidates. A data analysis agent, Finch, writes and runs Jupyter notebooks on raw data (flow cytometry, RNA sequencing). Robin uses o4-mini to synthesise literature and Claude 3.7 Sonnet as judge. The Methods say Robin "almost always called tools in the same order", so the paper's experiments were run as a fixed notebook sequence rather than free agent choice.
Full reading notes
Robin (Ghareeb et al., Nature, 2026)
What the paper does
A FutureHouse and Fordham team describes Robin, a workflow of language agents that, given a disease name, proposes in vitro assays and drug candidates and then analyses the laboratory data a human produces. Two literature agents built on PaperQA2, Crow (short reviews) and Falcon (long reports on each candidate), feed an LLM-judged pairwise tournament that ranks assays and candidates. A data analysis agent, Finch, writes and runs Jupyter notebooks on raw data (flow cytometry, RNA sequencing). Robin uses o4-mini to synthesise literature and Claude 3.7 Sonnet as judge. The Methods say Robin "almost always called tools in the same order", so the paper's experiments were run as a fixed notebook sequence rather than free agent choice.
It was asked for treatments for dry age-related macular degeneration (dAMD). Robin proposed boosting phagocytosis by retinal pigment epithelium (RPE) cells, a flow-cytometry assay, and 30 existing drugs. The human team tested five (exendin-4, fingolimod, MFGE8 as a known positive control, Y-27632, and AICAR plus TUDCA). Y-27632, a Rho kinase (ROCK) inhibitor, increased phagocytosis. Robin then proposed RNA-seq, which showed upregulation of ABCA1, and a second candidate round, which led to ripasudil (another ROCK inhibitor, approved for glaucoma in Japan) and KL001.
Results worth quoting
How results were checked, and by whom
- Bench experiments (external to the agents). Done by the authors' laboratory: ARPE-19 cells first (n = 3 wells), then primary RPE stem cells from one donor over 60 (four technical replicates), with fluorescent bovine rod outer segments instead of beads. Ripasudil raised phagocytosis 1.89-fold over DMSO in ARPE-19 by Finch's analysis (human analysis 1.75-fold; Fig. 4b). In primary cells ripasudil and Y-27632 were again hits and ripasudil more potent (Fig. 4d,e). KL001 was a hit in the primary cells. ABCA1 upregulation was repeated by RNA-seq in primary cells (Supplementary Fig. 17). No animal or clinical work, and ABCA1 was not tested functionally.
- Human reanalysis of Finch. A human scientist reanalysed the flow data and the RNA-seq results; the text says these "confirmed" Finch (Supplementary Figs 13–16; not read). Read alignment for RNA-seq was done by a human; Finch did the differential expression.
- Hallucinated references (Extended Data Fig. 4). One scientist, blinded to the source, tried to find every cited paper in ten randomly chosen proposals; an unidentifiable reference counted as hallucinated. Replacing Crow and Falcon with o4-mini alone sharply raised hallucinated references. For assay proposals, 44.5 ± 6.37% (mean ± s.e.m., n = 15) of o4-mini's references were hallucinated, against none from Crow. Ablating Crow alone did not raise the rate for drug-candidate reports, because Falcon corrected Crow's invented references.
- LLM judge versus experts. The judge's top ten matched the experts' top ten on 7.25 items on average, and repeated the same choice in 88% of repeated comparisons against 61% for human experts (Methods). The judge's prompt was itself written by Gemini 2.5 Pro from expert comparisons.
- Finch benchmarks. On 170 BixBench questions (n = 3 runs) Finch scored 22.8 ± 1.7% overall against 1.6 ± 1.2% for the base model with no data or tools; 47.9% on biostatistics, 15.3% on bioinformatics. On the paper's own round-two tasks, against expert rubrics, Finch scored 86 ± 0% for RNA-seq and 100 ± 0% for flow cytometry (n = 3 runs). Those tasks had task-specific prompts that the authors say they "thoroughly explored", so these two scores say little about unseen analyses (my reading).
- Comparison with OpenAI Deep Research. Given the same prompt, it produced 17 unique candidates; none was a hit in the primary-cell assay and none was a ROCK inhibitor (Extended Data Fig. 6). One run of one baseline.
Counts. 151 papers read to propose 10 mechanisms; about 400 papers for assay and candidate review; 30 candidates, of which five were tested in round one and ten in round two. Whether the other drugs in each round were hits is not stated in the text I extracted (Fig. 4b marks significance per drug). The Deep Research comparison says 19 Robin candidates in total; five plus ten is 15, so the count of 19 is unexplained (probably candidates added in later steps; uncertain).
What the humans did. Chose the disease; reviewed the ranked list and picked the top candidates; ran every experiment; chose the assay materials (used pHrodo beads and ARPE-19 cells although Robin suggested outer segments and primary or stem-cell RPE); picked drug concentrations from the literature (the highest proposed); chose which analysis to ask for ("flow cytometry", "differential expression analysis"); did RNA-seq alignment; chose which round-two drugs to test; and formatted the figures for publication. The abstract says "All hypotheses, experimental directions, data analyses and data figures in the main text of this report were produced by Robin"; the paper also says "The cycle continues until a human has a satisfactory drug candidate". Time claims (359–424 human hours against under 2 h; "200-fold") come from survey-based estimates, not measured human work (Table 1). Cost per Robin run was US$10.76.
The authors' own caveats
- The system needs expert prompt engineering: "Finch ... is also reliant on prompt engineering by domain experts". Finch's analyses "can vary between runs, even when given identical prompts and data"; the authors run eight trajectories and take a consensus (the same genes called in over 50% of trajectories, Fig. 3c).
- Robin gives experiment outlines, not executable protocols.
- Finch struggles on multi-step pipelines (15.3% on bioinformatics questions).
- All validation is in vitro; ripasudil needs in vivo work and ultimately a randomised trial. ROCK inhibitors were already known to enhance RPE phagocytosis (refs 35–37, including work by a co-author's laboratory); the authors describe Robin as targeting "low-hanging fruit".
- Reproducibility: Robin and Finch code, sample trajectories and RNA-seq data (SRA PRJNA1464762) are public. A tension: the reporting summary says "No data was excluded", while Fig. 4c and 4e label outliers (one in c, several in e) that were left out of the fits.
- Competing interests: most authors hold shares in Edison Scientific, a FutureHouse spin-out.
Ideas that travel beyond drug repurposing
- Output must be checkable (supports). Each recommendation is a specific drug and a specific assay; every literature claim has a reference that can be looked up, and the authors did look them up. The hallucination figures show the check mattering: an unaided model invented nearly half its references.
- External checks the agent cannot alter (supports, with qualification). The cell experiments are external, run by people. But human selection came between the agent's list and the bench, the agent's analysis of those data was only partly independently re-run, and the agent-versus-agent comparisons (ablations, Crow/Falcon quality) are judged by another LLM, whose agreement with experts is moderate (7.25 of 10).
- "Done"/"verified" as claims (supports). Falcon corrected Crow's hallucinated references, and the paper reports no signal that this happened until the authors ablated Falcon to find out; a clean final report did not show the stage before it was clean. Finch's consensus across eight runs is agreement among runs of one model, not verification; agreement with the human analysis is the stronger evidence.
- Silent omission (supports). The text does not say how many of each tested set failed, nor what happened to the untested drugs (25 of 30 in round one). The ablation of one agent that hid the errors of another is a clear case of omission by correction.
- Councils and multiple models (mixed). Three model families appear: o4-mini generates, Claude judges, Gemini writes the judge's prompt. Eight Finch trajectories give a consensus. The judge shows lower variability than experts (88% against 61%), but consistency is not the same as correctness.
- Disclosure (complicated). Prompts, trajectories and code are public, which is strong. But "produced by Robin" is a stronger statement than the Methods support, given the listed human choices and the deviation from Robin's suggested assay materials. The human-versus-agent split is detailed in the Methods; the abstract does not mention it.
Limits for our purposes
One disease, one assay, one laboratory, one donor for the primary-cell step, and replicates that are wells, not independent experiments. The headline candidate belongs to a drug class already known to work on the assay, so novelty is mostly in the specific drug, not the mechanism. The time-saving numbers are modelled estimates. For research outside wet-lab biology, the useful pieces are the reference-checking ablation, the practice of re-running the analysis agent several times, and the repeated check against a human analysis of the same data.
Nature 655, 487–496 (page range taken from the running footers)
Accelerating scientific discovery with Co-Scientist
Gottweis et al. · doi:10.1038/s41586-026-10644-y
A Google group (with collaborators at Stanford, Houston Methodist, Imperial and Sequome) describes Co-Scientist, a multi-agent system built on Gemini 2.0 that takes a research goal written in natural language and returns ranked research hypotheses and proposals. Agents have separate jobs: Generation, Reflection (acts as peer reviewer, with web search), Ranking (runs an Elo tournament of pairwise "scientific debates"), Proximity (clusters similar ideas), Evolution (writes new hypotheses from the best ones; it does not edit existing ones) and Meta-review (summarises recurring review criticisms and appends them to the other agents' prompts). A Supervisor agent schedules them on a task queue. Scientists can supply ideas, reviews and feedback through a chat interface.
Full reading notes
Co-Scientist (Gottweis et al., Nature, 2026)
What the paper does
A Google group (with collaborators at Stanford, Houston Methodist, Imperial and Sequome) describes Co-Scientist, a multi-agent system built on Gemini 2.0 that takes a research goal written in natural language and returns ranked research hypotheses and proposals. Agents have separate jobs: Generation, Reflection (acts as peer reviewer, with web search), Ranking (runs an Elo tournament of pairwise "scientific debates"), Proximity (clusters similar ideas), Evolution (writes new hypotheses from the best ones; it does not edit existing ones) and Meta-review (summarises recurring review criticisms and appends them to the other agents' prompts). A Supervisor agent schedules them on a task queue. Scientists can supply ideas, reviews and feedback through a chat interface.
It was asked to do three things, all in biomedicine: propose repurposing candidates, and combinations, for acute myeloid leukaemia (AML); propose epigenetic targets for liver fibrosis; and explain how capsid-forming phage-inducible chromosomal islands (cf-PICIs) spread across bacterial species.
Results worth quoting
Computational evaluation (Fig. 2, Extended Data Figs 1–2).
- 203 research goals: Elo ratings of the system's hypotheses rise across ten time buckets (Fig. 2a). The Elo is produced by the system's own Ranking agent. Extended Data Fig. 1 states that it "is auto-evaluated and not based on independent ground truth".
- 15 expert-written goals (seven biomedical experts): Co-Scientist's Elo ends above Gemini 2.0 variants, o1, o3-mini-high, DeepSeek R1 and the experts' own "best guess" answers (Fig. 2b), again on the system's own Elo.
- 11 of those goals, blinded human experts: preference ranks Co-Scientist 2.36 against 2.45, 2.45 and 2.73 for the three other systems (my reading of Fig. 2c; which value belongs to which system is not in the extracted text). Novelty 3.64 and impact 3.09 out of 5. With four systems, 2.5 is the chance mean, so the margin is narrow. The authors write "These evaluations reflect subjective expert assessments, not objective ground truth" and call the evaluation small-scale.
- Four LLM judges also preferred Co-Scientist (Extended Data Fig. 2).
Wet-lab checks (Figs 3–4, Extended Data Figs 3–6). All in cell lines, biological triplicates (n = 3), unblinded.
- AML, existing evidence. Thirty top-ranked candidates went to oncologists, who chose five for testing: binimetinib, pacritinib, cerivastatin, pravastatin, dimethyl fumarate. Three inhibited viability. Pravastatin and dimethyl fumarate showed little or no effect on MOLM-13 (Extended Data Fig. 4).
- AML, single agents proposed without prior preclinical evidence. Experts chose three of the ranked list: nanvuranlat, KIRA6, leflunomide. Only KIRA6 was active in several lines (IC50 10 nM in KG-1a, 180 nM in the non-AML control TK6, 1,750 nM in MOLM-13). Nanvuranlat and leflunomide had limited effect on MOLM-13. The text calls this "without oversight", yet experts selected which three to test.
- AML combinations. Seven combinations tested in MOLM-13 and KG-1a. MOLM-13 responses "predominantly synergistic"; KG-1a a mixture of synergy and antagonism (Fig. 4, Extended Data Table 2).
- Liver fibrosis. Experts picked three top-ranked hypotheses; the system named three epigenetic modifiers and drugs; two showed significant anti-fibrotic activity in human hepatic organoids without toxicity, one of them vorinostat (already approved for another cancer). Details are in a separate paper (ref. 1) and the Supplement.
- cf-PICIs. The group that had made the unpublished finding set the question. The system's top-ranked hypothesis, that cf-PICIs interact with diverse phage tails to expand host range, matched the experimental result, "in just 2 days" against years of laboratory work. The match was judged by the same group; no outside blind assessment is described.
Not in the main text. I found no baseline for the wet-lab hit rate (for example random drugs or expert-picked drugs), and no count of hypotheses produced against those tested beyond the figures above.
What the humans did. Wrote the goals (under 1 h of a clinician's time for the AML prompt); reviewed the ranked output and chose what to test (about 3 h for AML); suggested constraints (2,300 approved drugs across 34 cancers); ran all experiments. The authors say experts' time at both ends was comparable across the three studies.
The authors' own caveats
- Validation "remains preliminary"; the cell-line experiments are "a viability check", not a replacement for preclinical and clinical assessment, and an in vitro result does not "guarantee in vivo efficacy or clinical success".
- Reliance on open-access literature may omit paywalled work and gives "a systemic lack of access to negative experimental results". Mixed-quality sources create "a risk of propagating erroneous or irreproducible findings". They propose agents that trace claims to figures or data in a source.
- Inherited limits of the underlying models: "imperfect factuality and the potential for hallucinations". Rates of hallucinated or wrong hypotheses are not given in the main text.
- Possible homogenising of research directions and reduced critical thinking; misuse without peer review could worsen the reproducibility crisis.
- The Methods flag the worked ALS example as possibly containing errors, to be "interpreted with utmost caution".
- Reproducibility: the code is not public ("deep integration ... with proprietary internal infrastructure"); prompts and pseudocode are in the Supplement; access is through an experimental programme. Most authors are Alphabet employees and the study was funded by Alphabet. The reporting summary says all in vitro and computational replications "were successful" and no data were excluded; that is about assay and run repeats, not about how many hypotheses worked.
- A small inconsistency worth knowing: the reporting summary says 11 goals were used for LLM-judge preference; the main text says 15 goals for LLM judges and 11 for human experts. Fig. 3e prints an IC50 of 0.01 µM with a 95% interval of 0.012–0.081 µM, which does not contain the point estimate (probably a typographical error; not resolved).
Ideas that travel beyond biomedicine
- Output must be checkable (supports). Every hypothesis comes with a testable proposal and the key claims were tested at the bench. The system's own review uses literature search: without it, the Reflection agent scored known-non-novel ideas at 6.14 out of 10 for novelty; with it, 2.38 (Methods, ablation analysis).
- External checks the agent cannot alter (complicated). The bench and the blinded experts are external. The headline "quality improves with compute" curves are not: the Ranking agent that produces the Elo is part of the system, and the Meta-review agent feeds its critique back into the generators. The wet-lab step also sat after a human selection step, so the agent could not alter which drugs were tested, but experts could.
- "Done"/"verified" as claims (supports). The system attaches self-generated novelty reviews to each hypothesis (for example for KIRA6, "moderate novelty"); the authors judge it "reasonably well calibrated" from one example. Deep verification splits a hypothesis into sub-assumptions and checks each; that is a self-check, and its error rate on real hypotheses is not reported.
- Silent omission (supports, strongly). The authors name the main omission themselves: negative results and paywalled papers are not visible to the system. Candidates that failed (pravastatin, dimethyl fumarate, nanvuranlat, leflunomide) are reported, which is good practice; the untested remainder of the ranked lists and the hypotheses discarded by the tournament are not described.
- Councils and multiple agents (supports, with small effects). The ablation numbers are modest: Evolution lifted GPQA precision from 70.9% to 75.4%; Meta-review lifted the AUC for predicting a correct answer from 0.521 to 0.597 on the authors' constructed set (0.629 to 0.634 on GPQA diamond). An AUC of 0.521 is close to chance. All agents are one model family (Gemini 2.0). The debate prompt in the Ranking agent reduced positional bias. Other vendors' models appear only as baselines and judges.
- Disclosure (partly). Authors disclose employment and funding, and the human time spent. They do not give the failure rate of the hypothesis stream, and the system is not available for others to rerun.
Limits for our purposes
Large-lab, proprietary system, biomedicine, and a single model family. The three real-world cases each involved senior domain scientists who chose goals and candidates, so the paper shows a hypothesis generator under expert supervision, not an autonomous researcher. The most striking claim (the two-day recapitulation) was judged by the group that already knew the answer. Little here transfers to qualitative or humanities research; the transferable parts are the tournament-and-review pattern, the human selection step and the authors' list of what the system cannot see.
Nature 651, 914–919
Towards end-to-end automation of AI research
Lu et al. · doi:10.1038/s41586-026-10265-5
A team from Sakana AI, the University of British Columbia, the Vector Institute and Oxford (FLAIR) describes The AI Scientist, a pipeline of language-model agents that proposes a machine-learning research idea, checks it against the literature (Semantic Scholar), writes and runs the experiments, plots the results, writes a LaTeX paper and then has it reviewed by a second component, The Automated Reviewer. It only does computational experiments in machine learning. There are two modes: a template-based one that extends human-supplied starter code, and a template-free one that writes its own code through a parallel tree search (Methods). The template-free version used OpenAI o3 for ideas and code critique, Claude Sonnet 4 for code, GPT-4o for figures and best-first node selection, and o4-mini for review. A full template-free run takes "from several hours to over 15 h".
Full reading notes
The AI Scientist (Lu et al., Nature, 2026)
What the paper does
A team from Sakana AI, the University of British Columbia, the Vector Institute and Oxford (FLAIR) describes The AI Scientist, a pipeline of language-model agents that proposes a machine-learning research idea, checks it against the literature (Semantic Scholar), writes and runs the experiments, plots the results, writes a LaTeX paper and then has it reviewed by a second component, The Automated Reviewer. It only does computational experiments in machine learning. There are two modes: a template-based one that extends human-supplied starter code, and a template-free one that writes its own code through a parallel tree search (Methods). The template-free version used OpenAI o3 for ideas and code critique, Claude Sonnet 4 for code, GPT-4o for figures and best-first node selection, and o4-mini for review. A full template-free run takes "from several hours to over 15 h".
The headline test (section "Human evaluation results"): three generated papers were submitted, blind, to the ICLR 2025 "I Can't Believe It's Not Better" workshop, with the organisers' consent and ethics approval. One of the three scored 6, 7 and 6 (mean 6.33), above the workshop's average acceptance threshold; it was withdrawn by pre-agreed protocol. The other two did not meet the bar.
Results worth quoting
How results were checked, and by whom
- Workshop reviewers (external, human). 43 papers were reviewed; the three AI submissions were among them. Reviewers were told some submissions were AI-generated but not which, and could opt out. The accepted-level paper "ranked among the top 45%" of submitted papers (Fig. 2 caption). The workshop accepted 70% of submissions, against 32% for the ICLR 2025 main conference (Limitations).
- The authors' own team (internal, human). They concluded one paper met the workshop bar and none met the main-conference bar (Supplementary section C.2, as summarised in the main text).
- The Automated Reviewer (internal, automated). An ensemble of five o4-mini reviews plus a meta-review in which the model takes the role of area chair (Methods). It was checked against ICLR decisions from OpenReview: balanced accuracy 0.69 ± 0.04 on 2017–2024 papers and 0.66 ± 0.03 on 2025 papers published after the model's knowledge cutoff, against 0.66 for human reviewers in the NeurIPS 2021 consistency study (Table 1). The authors note the paper pools differ, so the comparison "is not exact".
- Reading Table 1 more closely. The automated reviewer's accuracy (0.65, 0.63) is below the humans' (0.73), and its false positive rate is high: 0.45 (before cutoff) and 0.52 (after) against 0.17 for humans, with a false negative rate of 0.17 against 0.52. It rejects too few bad papers. The text reports it as "comparable" on balanced accuracy and F1 and does not discuss this.
- Scaling claims. Paper quality scored by the Automated Reviewer rises with model release date (Fig. 1b; R = 0.517; n = 6 points per template-free model, n = 3 template-based) and with search budget (Fig. 3c). These are scores from the system's own reviewer, not from external reviewers.
How many checked versus produced. Three papers were checked externally. The text says the team "manually filtered the most promising outputs at each stage" and that three were selected on topic fit, whether the code "correctly implemented the proposed idea and ran without errors", and formatting. The number generated before selection is not in the main text (Supplementary section A.4; not read). One in three selected papers passed, so the pass rate over everything generated is unknown but lower.
What the humans did. They chose the workshop and prompted the system with its theme; filtered outputs at each stage; picked the three submissions; and, per the author contributions, three authors (R.T.L., Y.Y., S.H.) read and validated AI-generated papers to select the submissions and checked the code implementations behind them. They also ran the organiser communication, the ethics approval and the pre-planned withdrawal. The paper states that each submitted paper's workflow was "performed without any human modification", which is true of the text and code but not of the selection and checking around them.
The authors' own caveats
- "Only one of three submissions was accepted"; the system "cannot yet meet the standards of top-tier publications nor even do so consistently for workshops".
- Failure modes they list: naive or underdeveloped ideas, incorrect implementations of the main idea, lack of methodological rigour, errors in experimental implementation, duplicated figures in main text and appendix, and "many types of hallucinations, such as inaccurate citations". Rates are in the Supplement, not the main text.
- The Automated Reviewer may be contaminated by training data: balanced accuracy fell from 69% to 66% after the cutoff, which the authors read as, at most, a minimal effect.
- Persistent problems named: systems being easily fooled and overconfidently wrong, which "would prevent us from reliably trusting the outputs of systems like The AI Scientist".
- Risks named: overwhelming peer review, inflating credentials, using others' ideas without credit, adding noise to the literature.
- Reproducibility: code for both versions is public (Apache 2.0). Fig. 1b and Fig. 3c give replicate counts. The three submitted papers and trajectories sit in the Supplement. Competing interests: most authors are Sakana AI employees or consultants; the principal investigator also has Google DeepMind and Vector Institute affiliations.
Ideas that travel beyond machine learning
- Output must be checkable (supports, with a gap). The reviewers and the Automated Reviewer assess the manuscript (the Automated Reviewer processes the PDF). The paper is only as checkable as its link to the code. The authors themselves read the code of candidate papers before submitting; that step is what made "the code correctly implemented the idea" a checked fact rather than a claim, and it was done by people.
- External checks the agent cannot alter (complicated). The workshop review is a genuine external check, with a pre-registered withdrawal so the result could not be used to publish. But it covered three hand-selected papers. The check used to compare configurations and claim improvement over time (Fig. 1b, 3c) is the system's own Automated Reviewer, built by the same team, with a high false positive rate. Inside the pipeline, the status "non-buggy" means the code ran and a vision model found no problem with the plots; the authors still report incorrectly implemented ideas.
- "Done" and "verified" as claims (supports). The write-up is generated from the agent's notes and the system "automatically corrects any compilation errors". A paper that compiles, cites and has plots is not shown to be correct. The paper's own failure list (duplicated figures, invented citations) is a list of confident outputs that were wrong.
- Silent omission (supports). Selection of three from an unreported number, plus the stage-by-stage filtering, is invisible in the headline "passed peer review" result. The authors argue the papers "would still have been produced" without filtering; that is about cost, not about the pass rate.
- Councils and multiple models (supports, with a caveat). Different model families fill different roles (o3, Sonnet 4, GPT-4o, o4-mini). The five-review ensemble and area-chair meta-review use one model, so the five reviews share its biases; the paper treats the ensemble as improving robustness and does not test independence.
- Disclosure (supports). The study had consent of the venue and ethics approval, told reviewers that some papers were AI-generated, and withdrew all submissions "to avoid setting a precedent for publishing fully automated research before the scientific community has established clear standards for disclosure and evaluation". The reviewers knew the share but not which papers, and the co-authors who chose the papers were not blind.
Limits for our purposes
Machine learning only, with experiments that run entirely on a computer and can be re-run cheaply; the reproducibility of a research claim is therefore far easier here than in most fields. The central result is one paper in one workshop at a 70% acceptance rate, so it is an existence demonstration, not a rate. Several of the checks that would matter most to a sceptical reader (failure-mode frequencies, how many papers were produced, the selection protocol) are in the Supplement, which this note does not cover. The scaling claims rest on the system's own reviewer.
20 studies and benchmarks in which agents or language models tried to reproduce or replicate published research, newest first, with outcomes labelled. Benchmarks measure different things (rerunning supplied code, rebuilding code from the paper, re-analysing data), so the scores are not comparable with each other.
economics · replication of others
Mixed
LLM workflow run over 4,452 economics replication packages
Matthew Schwartz and colleagues
Discrepancies were flagged in 3,460 articles or their appendices.
Details
What was attempted
The authors describe an open-source workflow that gives an LLM a published article's replication package and asks it to do three things: reproduce the original calculations and flag discrepancies from the published findings, run automated sensitivity analyses, improve the calculations with a different implementation or algorithm, and develop extensions that fit the article's assumptions but are not in it. It was run across 4,452 replication packages from five economics journals. The date above is the NBER issue month (September 2026), with the day set to the first.
What happened
- Discrepancies were flagged in 3,460 articles or their appendices.
- Computation time was cut by more than a factor of ten in 496 articles, with similar or greater accuracy.
- Extensions were developed for 923 articles.
- The NBER page does not report how many packages reproduced cleanly.
Why it failed or succeeded, per the authors
Not stated on the page read; no failure analysis was available in the abstract. The page states that the LLMs were used in the analysis and writing, and that one author worked as a contractor for Anthropic.
Checks used
The abstract does not say whether flagged discrepancies were audited by a human. The working paper itself was not read.
Limits of this case
Abstract-level record only. A flagged discrepancy is not a confirmed error. Without the full paper it is unclear how many flags are real, how many reflect missing materials, and which model was used. Treat the counts as unverified until the paper is read.
Latest flagship models when written: Claude Fable 5.1 and GPT-5.6 Sol. Latest now: Claude Opus 5.5 and GPT-6.1 Sol.
scientific machine learning (PINNs, SINDy, physics-informed field theory) · replication of others
Mixed
Coding agents replicating four scientific machine learning papers with an evidence workflow
Atharva Hans and colleagues
All twelve workspaces passed the completion gate and all 158 recorded targets were matched with report coverage.
Details
What was attempted
The authors wrote a workflow, packaged as a coding-agent skill, in which the agent records each computational claim of a paper as a target, rebuilds the method, runs experiments, and links outputs to provenance and to the paper's claim. Completion has to pass external validation checks, not the agent's own statement. They ran three independent replications from paper materials alone, with author code excluded, on each of four papers (PIFT, PINN-I, PINN-II, SINDy), twelve runs in all. At least one of the four papers (PIFT) is co-authored by one of the authors, so this is a mixed case of own and others' work; authorship of the other three was not checked.
What happened
- All twelve workspaces passed the completion gate and all 158 recorded targets were matched with report coverage.
- Of 39 scalar anchor-run observations (13 standardised anchors, three runs each), 37 fell inside the paper-anchored threshold; two did not (a Schrödinger relative L2 error of 4.8 × 10^-2 against a 1% threshold, and a Navier–Stokes coefficient error of 16.4% against 10%).
- Elapsed time ranged from 1.2 to 13.0 hours; posterior median 6.9 hours for PINN-II and 5.0 for PINN-I against 2.2 for PIFT and 1.9 for SINDy.
- Reproduced PINN tables kept the reported accuracy patterns, but individual cell values did not match every printed digit.
- For a cylinder-wake target the paper materials omit the simulation trajectories and preprocessing, so the matched evidence supports the wake structure and trajectory geometry, not exact coefficient equality.
Why it failed or succeeded, per the authors
- The authors say a plain prompt does not reliably preserve progress or check that generated evidence supports claims, which the external gate fixes.
- "repeated runs differ in how papers are divided into targets, in numerical fidelity", and in effort and accepted evidence rules, even when all pass.
- Papers without seeds, sampled training sets or preprocessing details limit exact replication.
Checks used
- Workspace validation gate with hashes and provenance for every matched target.
- A separate paper-anchored scalar threshold analysis, independent of the per-target acceptance rule.
- Three independent runs per paper.
Limits of this case
Four papers and one agent (Codex with GPT-5.4); "matched" means the target-specific acceptance rule was met, which can be looser than the paper-anchored threshold. The authors' own workflow is being evaluated on a small set including at least one paper by an author.
Latest flagship models when written: Claude Fable 5 and GPT-5.5. Latest now: Claude Opus 5.5 and GPT-6.1 Sol.
psychology, political science, economics · replication of others
Partly succeeded
LLM reanalysis of 180 social and behavioural science studies
Tobias Holtdirk and colleagues
For 11 of the 180 studies the pipeline could not produce a usable effect size.
Details
What was attempted
An agentic pipeline received each study's original dataset, one focal statistical claim and, in the main condition, the relevant text of the article. It then wrote and ran its own analysis code, five runs per study, and the result was converted to a Cohen's d. The corpus was 180 published studies with predefined claims. For a subset the authors also compared the LLM with human reanalyses from the Multi100 collaboration. Three information conditions were tested: full paper, paper without methods, and abstract only.
What happened
- For 11 of the 180 studies the pipeline could not produce a usable effect size.
- For the rest, the LLM reached the same qualitative conclusion as the original in 80% of cases, and recovered the original effect size within ±0.05 in Cohen's d in 24%.
- In the subset with human reanalyses: same conclusion 95% for the LLM against 83% for human analysts; effect size within ±0.05 in 40% against 28%.
- Information condition barely mattered: within ±0.05 in 24% (full paper), 24% (no methods) and 22% (abstract only); within ±0.20 in 50%, 52% and 46%.
- GPT-5.5 and GLM-5.1 were run as robustness checks on the same task.
Why it failed or succeeded, per the authors
- The authors say the human comparison is descriptive: human analysts may have chosen their own defensible specifications, so higher LLM agreement could reflect a narrower, paper-anchored path rather than greater accuracy.
- Different defensible analytic choices can move effect sizes, in both humans and the LLM.
- They conclude "LLMs can support systematic audits of empirical results rather than substitute expert judgment".
Checks used
- Comparison against the original effect size at two tolerances (±0.05 and ±0.20) and on qualitative verdict (supports, opposite, inconclusive).
- Five independent runs per study; three information conditions; two additional models.
- Human reanalysis effect sizes from Multi100 for a subset.
Limits of this case
The study is a reproduction from original data with the claim handed to the model, so success is easier than a blind replication; and the strict tolerance rate of 24% shows exact recovery of effect sizes is rare. Only part of the supplement was read.
Latest flagship models when written: Claude Fable 5 and GPT-5.5. Latest now: Claude Opus 5.5 and GPT-6.1 Sol.
social sciences (four disciplines, 13 domains) · benchmark
Mixed
SocSci-Repro-Bench: Claude Code and Codex reproducing 54 social-science papers
Meysam Alizadeh and colleagues
Task-level accuracy: Claude Code 93.4%, Codex 62.1%. Paper-level (all tasks correct): 78.0% against 35.8%.
Details
What was attempted
The benchmark has 221 tasks from 54 papers in four social-science disciplines. Each paper's results are either fully reproducible with the supplied materials or demonstrably not, because data or code are missing (10 such tasks). Two coding agents, Claude Code and Codex, were run in sandboxes on the original replication materials. The authors also tested giving the paper PDF, and a prompt that nudged agents to match reported results.
What happened
- Task-level accuracy: Claude Code 93.4%, Codex 62.1%. Paper-level (all tasks correct): 78.0% against 35.8%.
- Both agents scored 100% on the non-reproducible tasks, correctly flagging missing data or code.
- Even excluding tasks where Codex produced nothing, 24.5% of its tasks were still wrong, and more than half of its papers had at least one wrong answer.
- Adding the paper PDF gave small gains (Claude Code from 93.4% to 94.5%).
- With a confirmatory nudge, accuracy on non-reproducible tasks fell from 100% to 70% for Claude Code and from 90% to 60% for Codex: the agents fabricated plausible outputs using numbers from the paper PDF instead of reporting that data were missing.
Why it failed or succeeded, per the authors
- Codex struggled with non-portable replication code: missing dependencies, hard-coded paths and incomplete packages.
- The authors read the gap as differences in agentic capability, not memorisation: Codex reports "Unknown" metadata for 92.6% of papers yet still scores 62.1%.
- They conclude "agents can be nudged toward confirmatory specification search through subtle prompt framing".
Checks used
- Papers chosen so the correct answer is known (fully reproducible or demonstrably not).
- Multiple runs: Claude Code ranged 92.6 to 94.5% and Codex 58.4 to 65.3% across runs.
- Memorisation probes (metadata recall) and a cross-model comparison.
Limits of this case
Two agents on one benchmark built by the same team; the authors compare their rates with lower earlier ones for general-purpose agents, but the benchmarks differ, so those comparisons are not like-for-like.
Latest flagship models when written: Claude Fable 5 and GPT-5.5. Latest now: Claude Opus 5.5 and GPT-6.1 Sol.
economics and political science · replication of others
Partly succeeded
Reproducing 48 social-science papers from the methods description alone
Benjamin Kohler and colleagues
Three best agents matched the sign of the original coefficient about 85 to 91% of the time.
Details
What was attempted
The authors asked whether agents can reproduce results without the original code. A system extracts a structured description of each paper's methods, masks the reported numbers, and gives an agent only that description and the original data. Agents never see the original code, results or paper. Four agent scaffolds and four LLMs were run on 48 papers that the Institute for Replication had verified as reproducible. Outputs were compared cell by cell with the originals, and an LLM traced each discrepancy to a root cause.
What happened
- Three best agents matched the sign of the original coefficient about 85 to 91% of the time.
- The best agent (GPT-5.4 on OpenCode) was within 1.96 original standard errors over 80% of the time; it beat Claude Code (Opus 4.6), GPT-5.3 Codex and GLM-5.
- Scaffold mattered: GPT-5.4 on OpenCode beat GPT-5.4 on Codex CLI and on mini-SWE-Agent, with much greater token use.
- Root-cause labels: most divergences were attributed to paper under-specification, fewer to agent errors, with the share of agent errors lower for stronger agents.
Why it failed or succeeded, per the authors
- Errors come mostly from papers under-specifying their methods and, to a lesser extent, from agents misreading or ignoring the method description. The abstract says "agents can largely recover published results".
- Better results came with more token use and longer analysis runs.
Checks used
- Strict information isolation between the agent and the original code and results.
- Deterministic, cell-level comparison against the published tables.
- A human audit of the LLM root-cause step: of 53 flagged divergences, 49 confirmed as meaningful (92%), but only 33 of 49 error-source labels confirmed (67%), 10 unsure. A different LLM judge changed the agent-error share.
Limits of this case
Papers were pre-selected as verified-reproducible and the data were given, so this tests re-implementation, not discovery of data. Root-cause shares come from an LLM judge and are indicative only, as the authors say.
Latest flagship models when written: Claude Opus 4.7 and GPT-5.5. Latest now: Claude Opus 5.5 and GPT-6.1 Sol.
economics and political science (quantitative social science) · replication game
Mixed
Institute for Replication AI replication games: human-only, AI-assisted and AI-led teams
Abel Brodeur and colleagues
Reproduction rate: human-only 94% (31 of 33 teams), AI-assisted 91% (32 of 35), AI-led 37% (13 of 35).
Details
What was attempted
In randomised replication games, 288 researchers formed 103 teams in three arms: human-only, AI-assisted (ChatGPT as a collaborator) and AI-led (ChatGPT drives, with minimal human oversight). Each team had seven hours to computationally reproduce a few pre-selected results from a published paper, find coding errors, and propose robustness checks. Twelve studies were used across events. Some papers contained coding errors the organisers had found earlier and not published. The published date above is the compile date of the accepted PNAS manuscript, not the journal issue date.
What happened
- Reproduction rate: human-only 94% (31 of 33 teams), AI-assisted 91% (32 of 35), AI-led 37% (13 of 35).
- Major coding errors found per team: human-only 1.70, AI-assisted 0.74, AI-led 0.23.
- Minor errors: 1.00, 1.40 and 0.69; differences were not statistically significant.
- AI-led teams proposed weaker robustness checks, six supplied none rated as good, and took longer.
Why it failed or succeeded, per the authors
- Participants in AI-led teams reported prompt fatigue, hallucinated reasoning paths, truncated context windows and long debugging loops.
- AI-assisted teams outsourced routine tasks such as boilerplate code and file paths, and kept conceptual control.
- The authors say the AI-led arm is probably a lower bound, since better prompting, newer models and supervision could raise it.
Checks used
- Randomised assignment to arms, with a one-hour ChatGPT training for the AI arms.
- Reproduction counted only if the team reproduced the pre-selected results.
- Transcripts of ChatGPT use retained, plus focus-group material for explanation.
- The authors conclude that "expert human judgment currently remains indispensable for reliable empirical verification".
Limits of this case
Only ChatGPT, only seven hours, and a small non-random set of twelve studies. The models used were 2024-era GPT-4 and GPT-4o; later agentic tools do much better on similar tasks (see socsci-repro-bench.md).
Latest flagship models when written: Claude Opus 4.6 and GPT-5.4. Latest now: Claude Opus 5.5 and GPT-6.1 Sol.
political science · replication of others
Succeeded
AI-assisted full-paper replication of 384 political science studies
Yiqing Xu (Stanford University) and colleagues
Fully or largely reproducible: 82.5% of papers after DA-RT adoption against 20.8% before; given an accessible replication package, 92.1% (234 of 254).
Details
What was attempted
The authors built a workflow that retrieves a paper's replication materials, rebuilds the computing environment, runs the code, and matches outputs to point estimates in the regression tables. They defined all empirical papers in three top political science journals from 2010 to 2025 and drew a stratified sample of 384 studies (8 per journal-year), covering 3,523 models. A second application re-ran 84 instrumental-variable studies and extracted 597 IV specifications for diagnostics. The numbers below are from version 3 of the preprint (31 May 2026); earlier versions reported different figures.
What happened
- Fully or largely reproducible: 82.5% of papers after DA-RT adoption against 20.8% before; given an accessible replication package, 92.1% (234 of 254).
- Of the 254 with accessible materials, 193 (76.0%) were fully reproducible, meaning every reported coefficient matched exactly, and 41 largely reproducible (more than 80% of coefficients matched).
- 191 papers were not fully reproducible; 130 of them lacked a package or had restricted data or code.
- Among the 61 papers with accessible materials below the strict threshold, 10 were partially and 10 not reproducible.
- IV corpus: all main-text regressions replicated in each of 84 papers, 1,910 models.
Why it failed or succeeded, per the authors
- "Failures trace almost entirely to missing or restricted data, not to code quality."
- Repeated issues in package structure were encoded as rules and version-controlled across runs, so coverage improved over versions.
- Journal verification requirements plus data-archiving mandates drive reproducibility, not the AI tool.
Checks used
- Coefficient-level matching against the published tables, with four verdict categories.
- Human review of failures; benchmark verification against a prior IV corpus (Lal et al. 2024).
- One IV paper dropped because table extraction produced no coefficients.
Limits of this case
It re-executes the authors' own code on their data, which is easier than re-implementing; it cannot rescue missing packages. Strong-sharing journals only. Read the abstract, introduction, results and supplementary tables, not every appendix.
Older account (before March 2026). Latest flagship models when written: Claude Opus 4.6 and GPT-5.2. Latest now: Claude Opus 5.5 and GPT-6.1 Sol. Agent tools have also changed since, so check whether a limitation reported here still applies.
social and behavioural science · benchmark
Partly succeeded
ReplicatorBench: agents replicating social and behavioural science claims with new data
Bang Nguyen and colleagues
Judging replicable versus non-replicable (version 3 numbers): GPT-5 accuracy 76.9% (macro F1 76.9), GPT-4o 61.5%, GPT-5-mini 48.7% (macro F1 47.9).
Details
What was attempted
Unlike code-rerun benchmarks, this one asks agents to replicate a claim: find and retrieve new data, design the analysis, run it, and judge whether the claim replicates. It has 19 observational studies from the Center for Open Science's SCORE project, including human-verified non-replicable ones. Agents are scored on about 1,568 rubric checkpoints across extraction and data retrieval, design and execution, and interpretation. The authors tested their ReplicatorAgent with four LLMs and varied the programming language and level of code access.
What happened
- Judging replicable versus non-replicable (version 3 numbers): GPT-5 accuracy 76.9% (macro F1 76.9), GPT-4o 61.5%, GPT-5-mini 48.7% (macro F1 47.9).
- Data retrieval was the weak stage: macro F1 about 8 to 19 across settings, and the agent found all of a study's data sources (hit@all) in only 5 to 16% of studies.
- GPT-5 and o3 had far fewer input-data errors than GPT-4o in execution (6 against 211 for GPT-5 and GPT-4o).
Why it failed or succeeded, per the authors
- Agents "struggle with retrieving new data, necessary to replicate a claim", while designing and running analyses went well.
- GPT-5-mini failed more on syntax and library use than on data handling.
- Stronger models also put more effort into execution than design, which the authors do not count as a failure.
Checks used
- Ground truth from human-verified replicable and non-replicable outcomes, with listed data-source URLs for the retrieval task.
- LLM-based scoring of stages (LLMEval), with 95% confidence intervals across runs; failures counted by category from run logs.
Limits of this case
Only 19 studies, all drawing on web-retrievable data; the authors note that primary data collection is out of scope. Read via the main text and tables, not every appendix.
Older account (before March 2026). Latest flagship models when written: Claude Opus 4.6 and GPT-5.2. Latest now: Claude Opus 5.5 and GPT-6.1 Sol. Agent tools have also changed since, so check whether a limitation reported here still applies.
political communication / computational text analysis · replication of others
Partly succeeded
Scott Cunningham: Claude Code replicating a PNAS text-classification paper with GPT-4o-mini
Scott Cunningham (Baylor University) and colleagues
The LLM labels agreed with the original classifier's labels in 69% of cases.
Details
What was attempted
Cunningham asked Claude Code to replicate the text-classification part of Card et al.'s PNAS paper on 140 years of immigration rhetoric in congressional and presidential speeches. The original used a fine-tuned RoBERTa classifier. He replaced it with zero-shot labels from gpt-4o-mini through the OpenAI Batch API. Claude Code handled the infrastructure, found and downloaded the original replication data, and ran the pipeline over about 305,000 speech segments. The post is part two of a pair; part one described the setup.
What happened
- The LLM labels agreed with the original classifier's labels in 69% of cases.
- The headline substantive findings held: Democrats and Republicans diverge sharply from the 1970s onward under both classifiers, and the country-of-origin ordering (Italy, China, Mexico) held.
- Direct polarity flips between pro and anti were about 4%; where classifiers disagreed the LLM mostly moved speeches to neutral (33% of pro and 44% of anti speeches reclassified as neutral).
- Run cost $10.99 and about 4.6 hours in total (2.6 hours of batch runtime, against 24 hours he had expected).
- He reports a lapse in his own protocol: a "Referee 2" review was run inside the same session, and a reader caught it.
Why it failed or succeeded, per the author
- Cunningham reads the move toward neutral as the LLM being cautious on ambiguous speeches.
- He argues the 69% figure should be read against modest agreement among human coders (Krippendorff's alpha 0.48).
Checks used
- Transition matrix between original and LLM labels, and comparison of the substantive conclusions across both classifiers.
- Human annotator agreement as a benchmark.
- No separate validation step is described in the excerpt read.
Limits of this case
The Substack page was paywalled after the summary, so the first-hand text read covers only the date, the title, the 69% agreement, the cost and the time; the other figures come from a third-party summary of the post (a wiki, velikov-mihail.github.io/ai-econ-wiki). It replicates a classifier-based result, not a full reproduction of the paper. A blog report, not peer reviewed.
Older account (before March 2026). Latest flagship models when written: Claude Opus 4.6 and GPT-5.2. Latest now: Claude Opus 5.5 and GPT-6.1 Sol. Agent tools have also changed since, so check whether a limitation reported here still applies.
radiation biology / genomics · audit of ai system
Mixed
Auditing three Kosmos radiation-biology claims against null models
Humza Nusrat and colleagues
DNA damage response versus p53 response (4 cell lines): Spearman rho -0.40, empirical p = 0.756; indistinguishable from random gene sets. Judged false.
Details
What was attempted
The authors took three hypotheses that Kosmos produced from public gene-expression datasets and re-ran each analysis themselves. Each claim was tested against an empirical null: 10,000 random five-gene sets, 10,000 random genes, or 5,000 random 12-gene signatures. The aim was to see which AI-generated claims survived a simple falsification test.
What happened
- DNA damage response versus p53 response (4 cell lines): Spearman rho -0.40, empirical p = 0.756; indistinguishable from random gene sets. Judged false.
- OGT and CDO1 predicting radiation-response modules (16 breast cancer lines): OGT r = 0.23, empirical p = 0.341, not supported. CDO1 r = 0.70, empirical p = 0.0039, a clear outlier. Judged well supported.
- 12-gene prostate recurrence signature (248 patients): c-index 0.613 beat random signatures (p = 0.0166) but the hazard ratio was not unusual (p = 0.3738). Judged plausible but uncertain.
- Overall verdict: one supported, one ambiguous, "one false result (DDR to p53)".
Why it failed or succeeded, per the authors
- The false claim rested on very small n and a fluent mechanistic story that hid weak evidence.
- The ambiguous signature probably captures proliferation, which many random gene sets also capture.
- The supported CDO1 claim had the largest sample (16 lines) and combined data patterns with literature.
- The authors argue AI outputs should be treated as hypotheses to falsify, and propose a four-step audit checklist.
Checks used
- Empirical null distributions from random gene sets with fixed seeds.
- Datasets and Kosmos reports public; the analysis code is "available upon request".
Limits of this case
Three claims from one system and one field, audited by the same authors who chose the tests; no second auditor. Not a replication of a published paper but a re-analysis of AI outputs.
Older account (before March 2026). Latest flagship models when written: Claude 4.1 Opus and GPT-5.1. Latest now: Claude Opus 5.5 and GPT-6.1 Sol. Agent tools have also changed since, so check whether a limitation reported here still applies.
metabolomics, materials science, neuroscience, statistical genetics · replication of others
Mixed
Kosmos reproducing unpublished and preprinted findings (developer-run)
Ludovico Mitchener and colleagues
Metabolomics run: nine of the top fifteen metabolites matched the original analysis exactly, with log10 fold changes closely matching (R2 = 0.998) and the same nucleotide-salvage conclusion.
Details
What was attempted
The developers gave Kosmos open-ended objectives and datasets, then checked whether its reports matched findings that were unpublished, preprinted or published after the underlying models' cut-off. Three of the seven highlighted discoveries are framed this way: a metabolomics dataset on cooling-induced neuroprotection (later preprinted), a perovskite solar-cell fabrication dataset, and neuron-morphology data from eight connectomes (matching a bioRxiv preprint). Separately, expert scientists audited 102 statements from three reports for accuracy.
What happened
- Metabolomics run: nine of the top fifteen metabolites matched the original analysis exactly, with log10 fold changes closely matching (R2 = 0.998) and the same nucleotide-salvage conclusion.
- Connectome run: Kosmos reproduced the two main quantitative results of the preprint (lognormal distributions; power-law relations between metrics).
- Statement audit: 79.4% of statements accurate overall; 85.5% of data-analysis statements reproducible, 82.1% of literature statements validated, and 57.9% of interpretive synthesis statements accurate.
- The seven highlighted discoveries include four novel ones that were not reproductions.
Why it failed or succeeded, per the authors
- Success is attributed to a shared world model linking data-analysis and literature agents over about 200 rollouts.
- Weakness is in interpretation: Kosmos tends to conflate statistical significance with scientific value, and invents unorthodox metrics.
- In the metabolomics run its extra claim about de novo synthesis could not be settled, because the data were steady-state levels only.
Checks used
- Expert scientists classified sampled statements as supported or refuted, without being told which came from which analysis type.
- Human-validation plots compared against Kosmos output, and the original studies as reference.
- "Independent scientists found 79.4% of statements in Kosmos reports to be accurate", per the abstract.
Limits of this case
Developer-run and selected: the seven highlighted cases were picked from collaborations, and failed or uninteresting runs are not itemised. See the independent audit of Kosmos in kosmos-radiation-biology-audit.md for a harsher picture.
Older account (before March 2026). Latest flagship models when written: Claude 4.1 Opus and GPT-5. Latest now: Claude Opus 5.5 and GPT-6.1 Sol. Agent tools have also changed since, so check whether a limitation reported here still applies.
astrophysics · benchmark
Failed
ReplicationBench: agents replicating astrophysics papers
Christine Ye and colleagues
Best average score was 22% (Claude 4.5 Sonnet). The abstract of the same version says best models score "under 20%", which does not match the 22% in the body text; the number depends on which table is read.
Details
What was attempted
The authors split 20 astrophysics papers into 111 tasks (version 2), each targeting a key result and co-developed with the original paper authors. Agents had to reproduce numerical results by re-implementing setups, derivations and analyses, working from a manuscript with the key numbers masked and with archival data. Seven frontier models were each run three times per paper. Domain experts also read agent traces to classify failures.
What happened
- Best average score was 22% (Claude 4.5 Sonnet). The abstract of the same version says best models score "under 20%", which does not match the 22% in the body text; the number depends on which table is read.
- Best-of-N scores were much higher than averages, so agents can do many tasks but not reliably.
- Median runtime of the best agent was over 12 times that of the worst.
- A no-computation check estimated answer memorisation at under 9% for every model.
- With the manuscript's key values left unmasked, some agents scored 15 to 20 points or more higher by copying values, usually after several failed attempts.
Why it failed or succeeded, per the authors
- Lack of persistence: agents submitted early, citing limits or impossible tasks, or filed fall-back answers and falsely claimed to have used "visual inspection".
- Conceptual or procedural errors: steps needing domain knowledge were left out, even when the procedure was spelled out.
- Technical execution failures: inefficient code on large data, legacy formats, loops of failed tool calls.
- Success was higher on tasks humans rated easier.
Checks used
- Tasks co-developed with original authors; expert annotation of agent traces.
- An LLM monitor to flag cheating and guessing, plus a no-computation baseline for memorisation.
- Ablation with unmasked manuscripts to measure copying.
Limits of this case
Replicates numerical results from public archival data; no new data collection. The authors say score variability can be about 5% and the memorisation rate is not a strict upper bound on contamination.
Older account (before March 2026). Latest flagship models when written: Claude 4.1 Opus and GPT-5. Latest now: Claude Opus 5.5 and GPT-6.1 Sol. Agent tools have also changed since, so check whether a limitation reported here still applies.
social psychology / computational social science · audit of ai system
Mixed
Analytic flexibility in silicon samples: re-examining Argyle et al. Study 3
Jamie Cummins (University of Bern and colleagues
Study 1: configurations varied substantially on all three criteria (recovering participant rankings, response distributions and between-scale correlations), and configurations that did well on one often did poorly on another.
Details
What was attempted
Silicon samples are LLM-generated datasets meant to stand in for human respondents. In Study 1 the author generated 252 silicon-sample configurations (varying model, sampling parameters, prompt format and demographic detail) for a controlled case using two social-psychological scales. In Study 2 he re-examined Study 3 of Argyle et al. (2023), a published silicon-sample use case, with 66 alternative configurations. The question was how far defensible choices change conclusions about how well silicon samples match human data.
What happened
- Study 1: configurations varied substantially on all three criteria (recovering participant rankings, response distributions and between-scale correlations), and configurations that did well on one often did poorly on another.
- Study 2: the correlation between human and silicon association structures ranged from r = .23 to r = .84 across the 66 configurations.
Why it failed or succeeded, per the authors
- Different defensible configuration choices can change conclusions about fidelity, so a single reported configuration may mislead.
- The author states that "different defensible configuration choices can materially alter conclusions about the fidelity of silicon samples".
- He calls for greater attention to analytic flexibility and outlines strategies researchers can use to reduce the threat.
Checks used
- Systematic variation of configurations against fixed human data.
- Three fidelity criteria for Study 1; association-structure correlation for Study 2.
Limits of this case
A methods audit of one published use case and one controlled case, not a replication of a human study; the absolute range depends on the configurations the author chose. Only the abstract and introduction were read closely; the numbers above are the abstract's.
Older account (before March 2026). Latest flagship models when written: Claude 4.1 Opus and GPT-5. Latest now: Claude Opus 5.5 and GPT-6.1 Sol. Agent tools have also changed since, so check whether a limitation reported here still applies.
economics and political science · benchmark
Failed
REPRO-Bench: agents assessing reproducibility of 112 social-science papers
Chuxuan Hu and colleagues
Best existing agent (CORE-Agent with GPT-4o): 21.4% accuracy, below the 25% expected by guessing among four scores.
Details
What was attempted
Agents were given a social-science paper (PDF) and its reproduction package, and asked to rate its reproducibility from 1 (not reproducible) to 4 (fully reproducible). Ground truth was taken from published reproduction reports by social scientists: 112 task instances across Stata, R and Python. The task is to execute the package and check consistency with the paper's claims, not only to rerun the code.
What happened
- Best existing agent (CORE-Agent with GPT-4o): 21.4% accuracy, below the 25% expected by guessing among four scores.
- The authors' REPRO-Agent, with a four-phase template, a fallback score and few-shot examples of common errors, reached 36.6%, a 71% relative gain.
- The authors report that the three agents did about as well on packages with several data formats (52% average accuracy) as on single-format packages (54%).
Why it failed or succeeded, per the authors
- Four failure types: (1) wrong comparison of reproduced and reported results; (2) Stata scripts whose errors go to log files, not the terminal, so the agent misreads silence; (3) failed library installation; (4) files not in the execution directory, the most frequent type.
- Agents often concluded data were missing without searching the package.
- Reading whole code files instead of relevant sections made inconsistency detection hard.
Checks used
- Human reproduction reports as ground truth, annotated by the authors in a consensus process.
- Accuracy measured on the 1–4 score.
Limits of this case
All agents used one 2024 model (GPT-4o); the later SocSci-Repro-Bench and Kohler et al. results (see socsci-repro-bench.md, kohler-read-paper-write-code.md) show much higher rates with newer coding agents. The abstract reports an "accuracy of only 21.4%".
Older account (before March 2026). Latest flagship models when written: Claude 4 Opus. Latest now: Claude Opus 5.5 and GPT-6.1 Sol. Agent tools have also changed since, so check whether a limitation reported here still applies.
biomedical informatics / Alzheimer's disease · replication of others
Partly succeeded
GPT-4o agent team reproducing five Alzheimer's disease studies
Nic Dobbins and colleagues
On average the agents approximately reproduced 53.2% of findings per study; in 3 of 5 studies a majority of findings was approximately reproduced.
Details
What was attempted
The authors chose five highly cited papers that used the National Alzheimer's Coordinating Center "Quick Access" dataset and appeared reproducible from that dataset alone. A simulated research team of GPT-4o agents, built with AutoGen, wrote and executed code to reproduce each study's findings. The agents were given only the abstract, the methods section and the data dictionary. The authors extracted 35 key findings from the abstracts as the targets.
What happened
- On average the agents approximately reproduced 53.2% of findings per study; in 3 of 5 studies a majority of findings was approximately reproduced.
- Numeric values and range-based findings often differed from the originals, and the agents used statistical methods or parameters that varied from the originals, though overall trends and significance were sometimes similar.
- Python code failed in 26 of 55 execution attempts (47.2%), often because of wrong column names (for example "AGE" instead of "NACCAGE").
- Agent conversations ran 7 to 52 turns (mean 35.6).
Why it failed or succeeded, per the authors
- In some cases the agents reproduced the techniques and findings accurately, but "they failed because of implementation flaws or missing methodological information".
- Methods sections do not fully state details, and the agents sometimes approximated a method with a generic statistical test.
- The authors also name loss of context across long conversations and the risk of trusting agent-written code.
Checks used
- Authors compared agents' findings with the originals' abstract-level findings, one by one.
- Review of the agent transcripts and of sequences of speakers.
- Studies chosen in advance as reproducible with this dataset alone.
Limits of this case
Five studies, one dataset, one 2024-era model, and the authors call it exploratory. The authors note the original papers may be in GPT-4o's training data. The date is of the first arXiv version; the authors have revised the paper since.
Older account (before March 2026). Latest flagship models when written: Claude 4 Opus. Latest now: Claude Opus 5.5 and GPT-6.1 Sol. Agent tools have also changed since, so check whether a limitation reported here still applies.
machine learning · benchmark
Partly succeeded
PaperBench: agents replicating 20 ICML 2024 papers from scratch
Giulio Starace and colleagues
Best agent on the main benchmark: Claude 3.5 Sonnet (New) with a basic scaffold, average replication score 21.0%. OpenAI o1 scored 13.2%; the other models scored under 10%.
Details
What was attempted
Agents were given the text of 20 ICML 2024 Spotlight and Oral papers plus an addendum of clarifications. They had to write a codebase from scratch, run it, and produce a reproduce.sh entry point that regenerates the paper's empirical results. Each paper has a hierarchical rubric (8,316 gradable outcomes in total) that was co-developed with one of the original authors. Submissions were re-run and graded by an LLM judge against the rubric. The agents were barred from using the authors' own code repositories, which were blacklisted.
What happened
- Best agent on the main benchmark: Claude 3.5 Sonnet (New) with a basic scaffold, average replication score 21.0%. OpenAI o1 scored 13.2%; the other models scored under 10%.
- A variant scaffold that stops agents ending early (IterativeAgent) lifted o1-high to 24.4% and o3-mini-high to 8.5%, but lowered Claude 3.5 Sonnet to 16.1%. A 36-hour o1 run reached 26.0%.
- Code-Dev variant (code written but not executed or graded on results): o1 scored 43.4%.
- Human baseline: eight ML PhD students attempted a subset. The introduction says a 3-paper subset, section 5.4 says 4 papers; best-of-3 humans scored 41.4% after 48 hours against 26.6% for o1 on the same subset. o1 led in the first hours and humans overtook it after about 24 hours.
Why it failed or succeeded, per the authors
- Models other than Claude 3.5 Sonnet often stopped early, claiming they had finished or hit an unsolvable problem; "frequently finished early, claiming that they either had finished the entire replication or had faced a problem they couldn’t solve".
- All agents failed to plan how to use the limited time; o3-mini often struggled with tool use.
- The authors read this as weak long-horizon behaviour: good at writing plans, poor at carrying them out.
- Scaffold prompts that helped o-series models hurt Claude, so scores depend on scaffold as well as model.
Checks used
- Rubrics co-developed with original paper authors.
- LLM judge validated on a separate gold-label set graded by human experts (best judge F1 0.83).
- Three runs per paper; blacklist check for use of forbidden resources across 646 runs.
- Human expert baseline under similar conditions.
Limits of this case
Replicates ML code and experiments, where compute and code are the only barriers; the LLM judge is a stand-in for human grading. Headline numbers are from the first version of the paper held in Zotero.
Older account (before March 2026). Latest flagship models when written: . Latest now: Claude Opus 5.5 and GPT-6.1 Sol. Agent tools have also changed since, so check whether a limitation reported here still applies.
natural language processing · benchmark
Failed
SciReplicate-Bench: reproducing algorithms from 36 NLP papers
Yanzheng Xiang and colleagues
Best result: Claude 3.7 Sonnet with Sci-Reproducer, 39.0% execution accuracy; all models were below 40%.
Details
What was attempted
The benchmark has 100 tasks from 36 NLP papers published in 2024. Each task asks a model to write the code for one algorithm described in a paper, inside the authors' repository, with test cases from the original implementation. The authors built Sci-Reproducer, with a Paper Agent that reads the paper and related literature and a Code Agent that retrieves repository dependencies and writes the code. Several reasoning and non-reasoning LLMs were tested.
What happened
- Best result: Claude 3.7 Sonnet with Sci-Reproducer, 39.0% execution accuracy; all models were below 40%.
- Average execution accuracy across models with Sci-Reproducer was 0.235.
- Without the agent to read literature and repository context, most LLMs scored below 0.1 execution accuracy; Sci-Reproducer added 0.181 on average.
- When the authors added the missing or mismatched information from the papers as extra input, execution accuracy rose by 0.05 for GPT-4o-mini, 0.25 for DeepSeek-V3 and 0.04 for o3-mini-low.
Why it failed or succeeded, per the authors
- Papers omit hyperparameters, numerical-stability tricks, implementation logic and coding strategy; the authors conclude "missing or inconsistent algorithm descriptions as key barriers to successful reproduction".
- Models, like human developers, cannot see runtime information from a compiler in this setup, which limits debugging.
Checks used
- Execution against test cases from the original repositories.
- Reasoning-graph accuracy to score understanding, plus CodeBLEU and dependency recall.
- Authors' manual comparison of every task's code against its paper description to find missing or mismatched details.
Limits of this case
A code-completion task inside a given repository, not a reproduction of a paper's results; only the abstract, introduction, results and analysis sections were read closely. The aggregator claim about syntax-error reduction was not checked and is not used.
Older account (before March 2026). Latest flagship models when written: . Latest now: Claude Opus 5.5 and GPT-6.1 Sol. Agent tools have also changed since, so check whether a limitation reported here still applies.
machine learning / recommender systems · audit of ai system
Mixed
Independent evaluation of Sakana's AI Scientist
Joeran Beel and colleagues
Five of twelve proposed experiments (42%) failed to run because of coding errors; the system flagged 10 generated ideas and both seeds as novel, including well-established ones such as micro-batching for SGD.
Details
What was attempted
The authors installed the open-source AI Scientist and ran it end to end on recommender-systems research ideas, to test Sakana's claims that it automates the full research lifecycle. They supplied two seed ideas, generated further ideas, ran the experiments and read the resulting manuscripts. This is a test of a system's claims rather than of one published paper. The Zotero title is the first version's title; the arXiv v3 read here has a different title.
What happened
- Five of twelve proposed experiments (42%) failed to run because of coding errors; the system flagged 10 generated ideas and both seeds as novel, including well-established ones such as micro-batching for SGD.
- Experiments that ran often gave logically flawed results; in one, an energy-efficiency experiment reported accuracy gains while using more compute.
- Code edits were small, about 8% more characters in the first iteration, shrinking in later ones.
- Seven manuscripts were produced; median five citations each, five of 34 citations from 2020 or later, four of seven manuscripts with missing or misplaced figures or placeholder text, and four of seven with incorrect or hallucinated numbers.
- Cost was roughly $6 per manuscript in API charges.
Why it failed or succeeded, per the authors
- Novelty checking relies on keyword searches over Semantic Scholar, not synthesis of the literature.
- The system depends on the supplied templates and cannot critically assess its own results.
- The authors conclude it is a notable advance in producing complete manuscripts, with quality like that of a rushed undergraduate.
Checks used
- Manual reading of every generated manuscript and experiment log.
- Comparison against Sakana's stated claims.
- A containerised reproduction package published so others can rerun the evaluation.
Limits of this case
Small sample (twelve experiments, seven manuscripts) in one subfield, on one version of the tool. The headline claim, "five out of twelve proposed experiments (42%) failed due to coding errors", comes from that sample.
Older account (before March 2026). Latest flagship models when written: . Latest now: Claude Opus 5.5 and GPT-6.1 Sol. Agent tools have also changed since, so check whether a limitation reported here still applies.
computer science, social science, medicine · benchmark
Mixed
CORE-Bench: computational reproducibility from code and data, and its later 'solved' status
Zachary S. Siegel and colleagues
Best agent (CORE-Agent with GPT-4o): 60.0% Easy, 57.8% Medium, 21.5% Hard.
Details
What was attempted
CORE-Bench has 270 tasks drawn from 90 papers with public code, in Python and R. The agent must install dependencies, run the code, and answer questions about the outputs. There are three difficulty levels: results given (Easy), a run command given (Medium), or only a README (Hard). A task counts as solved if every answer falls within a prediction interval from three manual reproductions. The authors tested AutoGPT and their own CORE-Agent with GPT-4o and GPT-4o-mini.
What happened
- Best agent (CORE-Agent with GPT-4o): 60.0% Easy, 57.8% Medium, 21.5% Hard.
- Vision questions about figures were harder than written ones (59.3% against 87.9% on Easy, for GPT-4o).
- Later update, from the Princeton HAL team (3 December 2025): Claude Code with Claude Opus 4.5 scored 77.8% on Hard under automatic grading and 95.5% after manual validation; the CORE-Agent scaffold with the same model scored about 42%. They declared the benchmark solved and fixed grading errors in 8 tasks, and removed one task with a dead dataset URL.
Why it failed or succeeded, per the authors
- Easy failures: picking the wrong figure or file when outputs were spread across several.
- Hard failures: agents did not finish resolving dependency versions before hitting the cost limit and kept retrying the same install; raising the cost limit did not help.
- Of the 2025 update: the automatic grader penalised tiny floating-point differences, underspecified tasks and correct-but-different methods; the authors say scaffold choice alone moved Opus 4.5 from 42% to 78%.
Checks used
- Three manual reproductions per task to define the prediction interval.
- Three repeated agent runs with confidence intervals.
- For the 2025 update, manual re-grading of disagreements.
Limits of this case
The 2025 update comes from a leaderboard note and a social post by the HAL team, read via their pages (the post's full text was only partly visible), not from a paper. The benchmark checks that code reruns and outputs are read correctly, not whether the paper's claims are right.
Older account (before March 2026). Latest flagship models when written: . Latest now: Claude Opus 5.5 and GPT-6.1 Sol. Agent tools have also changed since, so check whether a limitation reported here still applies.
psychology and management · replication of others
Mixed
LLMs as simulated participants: replicating 156 psychology and management experiments
Ziyan Cui and colleagues
Main effects replicated at 72.7% (GPT-4), 80.6% (Claude) and 76.2% (DeepSeek); interaction effects at 45.7%, 61.7% and 62.9%.
Details
What was attempted
The authors reran 156 scenario-based experiments from top social-science journals, 2015 to 2024, with LLMs playing the participants. Each model saw only one experimental condition at a time, so it was blind to the design and hypothesis. The 156 studies (50 articles from five management and psychology journals, randomly selected) gave roughly 620 to 680 main effects and 130 to 165 interaction effects, depending on the model (DeepSeek could not process images, so its set was smaller). A replication counted if the LLM sample preserved the direction and statistical significance of the original effect.
What happened
- Main effects replicated at 72.7% (GPT-4), 80.6% (Claude) and 76.2% (DeepSeek); interaction effects at 45.7%, 61.7% and 62.9%.
- LLM effect sizes were much larger than human ones, with Fisher Z values about 2 to 3 times higher.
- Where original studies reported null findings, LLMs produced significant results 68 to 83% of the time.
- Replication rates were lower for race-related effects (GPT-4: 41.5% against 76.8%) and gender-related effects (57.8% against 74.3%). Claude showed no significant difference for race or gender.
- 35.3% of studies needed prompt adaptations to keep the models engaged with the manipulation.
Why it failed or succeeded, per the authors
- LLMs may produce cleaner data with less noise, which narrows confidence intervals but risks overestimating effects.
- Lower rates for socially sensitive topics suggest models do not capture culturally sensitive social reasoning.
- The authors say "LLMs show significantly lower replication rates for studies involving socially sensitive topics such as race, gender and ethics".
Checks used
- Rate of direction and significance preserved against the original published statistics.
- Comparison across three LLMs and across study features (journal, sample, platform, publication year, citations).
- A contamination argument: the papers were behind subscription journals, a preliminary test showed GPT-4 could not describe them accurately, and publication-year and openness effects were non-significant.
Limits of this case
"Replication" here means an LLM-simulated sample reproducing a human study's effect, not an agent re-running a study. It does not test what the effect would be on real people. The count of studies differs across versions and secondary sources (154 or 156 appear), and this record follows the version read.
Older account (before March 2026). Latest flagship models when written: . Latest now: Claude Opus 5.5 and GPT-6.1 Sol. Agent tools have also changed since, so check whether a limitation reported here still applies.