# GPT-4o agent team reproducing five Alzheimer's disease studies

- Outcome: Partly succeeded
- By: Nic Dobbins and colleagues
- Field: biomedical informatics / Alzheimer's disease
- Published: 2025-05-29
- Source: https://arxiv.org/abs/2505.23852

## What was attempted

The authors chose five highly cited papers that used the National Alzheimer's Coordinating Center "Quick Access" dataset and appeared reproducible from that dataset alone. A simulated research team of GPT-4o agents, built with AutoGen, wrote and executed code to reproduce each study's findings. The agents were given only the abstract, the methods section and the data dictionary. The authors extracted 35 key findings from the abstracts as the targets.

## What happened

- On average the agents approximately reproduced 53.2% of findings per study; in 3 of 5 studies a majority of findings was approximately reproduced.
- Numeric values and range-based findings often differed from the originals, and the agents used statistical methods or parameters that varied from the originals, though overall trends and significance were sometimes similar.
- Python code failed in 26 of 55 execution attempts (47.2%), often because of wrong column names (for example "AGE" instead of "NACCAGE").
- Agent conversations ran 7 to 52 turns (mean 35.6).

## Why it failed or succeeded, per the authors

- In some cases the agents reproduced the techniques and findings accurately, but "they failed because of implementation flaws or missing methodological information".
- Methods sections do not fully state details, and the agents sometimes approximated a method with a generic statistical test.
- The authors also name loss of context across long conversations and the risk of trusting agent-written code.

## Checks used

- Authors compared agents' findings with the originals' abstract-level findings, one by one.
- Review of the agent transcripts and of sequences of speakers.
- Studies chosen in advance as reproducible with this dataset alone.

## Limits of this case

Five studies, one dataset, one 2024-era model, and the authors call it exploratory. The authors note the original papers may be in GPT-4o's training data. The date is of the first arXiv version; the authors have revised the paper since.
