FailedBenchmark
SciReplicate-Bench: reproducing algorithms from 36 NLP papers
Also failed
What was attempted
The benchmark has 100 tasks from 36 NLP papers published in 2024. Each task asks a model to write the code for one algorithm described in a paper, inside the authors' repository, with test cases from the original implementation. The authors built Sci-Reproducer, with a Paper Agent that reads the paper and related literature and a Code Agent that retrieves repository dependencies and writes the code. Several reasoning and non-reasoning LLMs were tested.
What happened
- Best result: Claude 3.7 Sonnet with Sci-Reproducer, 39.0% execution accuracy; all models were below 40%.
- Average execution accuracy across models with Sci-Reproducer was 0.235.
- Without the agent to read literature and repository context, most LLMs scored below 0.1 execution accuracy; Sci-Reproducer added 0.181 on average.
- When the authors added the missing or mismatched information from the papers as extra input, execution accuracy rose by 0.05 for GPT-4o-mini, 0.25 for DeepSeek-V3 and 0.04 for o3-mini-low.
Why it failed or succeeded, per the authors
- Papers omit hyperparameters, numerical-stability tricks, implementation logic and coding strategy; the authors conclude "missing or inconsistent algorithm descriptions as key barriers to successful reproduction".
- Models, like human developers, cannot see runtime information from a compiler in this setup, which limits debugging.
Checks used
- Execution against test cases from the original repositories.
- Reasoning-graph accuracy to score understanding, plus CodeBLEU and dependency recall.
- Authors' manual comparison of every task's code against its paper description to find missing or mismatched details.
Limits of this case
A code-completion task inside a given repository, not a reproduction of a paper's results; only the abstract, introduction, results and analysis sections were read closely. The aggregator claim about syntax-error reduction was not checked and is not used.