2025-2026 multi-agent benchmarks and what each measures
the five canonical benchmarks
MultiAgentBench
(MARBLE)
topologies:
star/chain/tree/graph
graph wins research
coord-tax past ~4
ACL 2025
COMMA
multimodal,
asymmetric info
GPT-4o struggles
to beat random
null-result signal
MedAgentBoard
4 medical tasks
single-LLM often
matches multi-agent
over-engineering check
domain stress test
AgentArch
tools + memory +
orchestration
per-layer value
attribution
enterprise stack
SWE-bench Pro
1865 problems
~23% vs 70%+ on
Verified
contamination
gating 2026 test
read benchmark claims skeptically
[1]
which benchmark + split? Verified vs Pro is a 40-point gap for frontier models
[2]
contamination: is the benchmark post-training-cutoff?
[3]
baselines: vs single-LLM, vs random, vs prior multi-agent
[4]
statistical significance: N trials, CI, p-value
[5]
task diversity: one task or many — generalization matters
[6]
cost disclosure: tokens + wall-clock — 20x cost for 10% lift is a business decision