AI Scientist v2 — research loop, measured failure rates
idea
agentic tree search
novelty check
~25% mislabel
experiment
42% failure
(Beel 2025)
VLM figure critique
polishes visuals
masks 70%
writeup
review + submit
measured results (Yamada 2025, Sakana Nature 2026)
· v2-generated paper accepted at ICLR 2025 workshop (disclosed)
· template-free loop replaces v1's fixed scaffolding
· Nature 2026 paper documents end-to-end pipeline
· Beel et al. 2025 external eval flagged polish masking experiment flaws
operational safety (Sakana repo README, paraphrased)
- the codebase executes LLM-written code
- dangerous packages, uncontrolled web access, unintended processes possible
- Docker isolation recommended (seccomp / gVisor preferred for multi-day runs)
- disclose agent-authored papers; review before submission
- polish stage can mask experiment weakness; require separate audit
- evaluator is peer review, which is weak and noisy
unverified domain + full-text output surface + public venues = strongest controls in the phase