Three 2026 agent benchmarks: code, generalist, multi-environment
SWE-bench code
2,294 GitHub issues, 12 Python repos
agent gets: repo + issue text
produces: patch
FAIL_TO_PASS: now passes
PASS_TO_PASS: stays passing
Verified (OpenAI): 500 clean tasks
removes ambiguity, weak tests, unclear fixes
SWE-bench+: contamination audit
32.67% solution leakage; 31.08% weak tests
GAIA generalist
466 questions, 300 private leaderboard
simple for humans (92%)
hard for AI (GPT-4+plugins: 15%)
Level 1 single tool, simple
factual lookup
Level 2 multi-step reasoning
search + extract + compute
Level 3 long tool chain + multimodal
images, PDFs, chained web navigation
AgentBench multi-env
8 environments, ~4k-13k turns/split
code: Bash, DB, KG
games: Alfworld, LTP
web: WebShop, Mind2Web
open-ended generation
what benchmarks do not measure
operational cost: tokens, wall-clock, steps-to-resolution
safety behavior under adversarial conditions (Lesson 27)
performance on your domain (custom evals, Lesson 30)
tail failures (benchmarks average; operators care about the worst 1%)
always report both raw score AND Verified / +-audited score
single-number leaderboard fixation is the #1 anti-pattern. report distributions.