Thomas Piacentini
← Workshop

Cheat sheet

Agent Benchmarks: SWE-bench to OSWorld

Original text · English

Pick the benchmark, gate on the right tests, check contamination and compare runs on equal denominators.

Single-agent benchmarks side by side

FactorSWE-benchGAIAWebArenaOSWorld
TestsCode patchingGeneralist tool useWeb tasks, long horizonReal computer tasks
Size2,294 issues, 12 Python repos466 questions, 300 private812 tasks, 4 apps369 tasks, 3 OSes
Agent inputCodebase at pre-fix commit + issueQuestion. Web, multi-modalitySelf-hosted apps + utilities1920×1080 screenshots
EvaluatorFAIL_TO_PASS and PASS_TO_PASS testsNot givenExecution-based, gym APIsNot given
Best at releaseSWE-agent 12.5%GPT-4 with plugins 15%GPT-4 agent 14.41%Best model 12.24%
HumanNot given92%78.24%72.36%
Main trapContaminationConfusion with code benchmarksStale pinned app versionsGUI grounding
Sources

Multi-agent benchmarks side by side

FactorMARBLECOMMAMedAgentBoardSWE-bench Pro
MeasuresStar, chain, tree, graph topologiesMultimodal asymmetric information4 medical task categoriesUncontaminated, later cutoffs
ScoringMilestone KPIs, partial progressAgent-agent collaborationMulti-agent vs single-LLM vs rules1865 problems, 41 repos
HeadlineGraph best for researchFrontier models struggle vs randomMulti-agent does NOT dominate~23% on Pro vs 70%+ on Verified
LessonCognitive planning adds ~3%Measure before you claimMany systems are over-engineeredVerified alone is not evidence
Sources

Use X when

  • SWE-bench Verified: a code agent. Always report Verified scores.
  • SWE-bench Pro: a contamination reality check. Pro is the current gating test.
  • GAIA: a generalist agent. Use the private leaderboard split.
  • AgentBench: a multi-environment comparison across 8 environments.
  • WebArena Verified: a web agent. Self-host it for continuous evaluation.
  • OSWorld: a desktop agent. Run it in a VM fleet.
  • MARBLE: you compare coordination topologies apples-to-apples.
  • AgentArch: you must justify each layer of an enterprise stack.
  • Custom evals: your product flows. Capture gold trajectories for your top 20 tasks.
Sources

SWE-bench gate: run_task

  1. Harness copy state_before. Count passing FAIL_TO_PASS and PASS_TO_PASS checks.
  2. Agent apply the patch to a copy of the state.
  3. Harness count passing checks again on the new state.
  4. Fix gate all FAIL_TO_PASS tests must pass after the patch.
  5. No-regression gate ptp_broke must be 0.
  6. Result resolved is true only when both gates pass.
  7. Report resolution rate is resolved tasks over all tasks.
Sources

Harness Bench pipeline

casesNewModelContextModelanswerResultCases JSONID, Prompt, ExpectedstoreCases()strict decode, max 1000EvaluatePolicy()3 policies, call budgetprocessLeaderboard()equal denominatorsRecordingauthored_fixturestoreHTTPModel()chat-completionsmodelCorrect()exact answer contract
Nodes and connections
NodeRoleKind
Cases JSONID, Prompt, Expectedstore
Cases()strict decode, max 1000
EvaluatePolicy()3 policies, call budgetprocess
Leaderboard()equal denominators
Recordingauthored_fixturestore
HTTPModel()chat-completionsmodel
Correct()exact answer contract
FromToConnection
Cases JSONCases()
Cases()EvaluatePolicy()cases
RecordingEvaluatePolicy()NewModel
HTTPModel()EvaluatePolicy()ContextModel
EvaluatePolicy()Correct()answer
EvaluatePolicy()Leaderboard()Result
Sources

Run the lessons and the project

CommandWhat it does
python3 code/main.pyRuns the toy harness of lessons 14.19, 14.20 and 16.24
python3 scripts/project_test.py harness-bench --init /tmp/harness-bench-workCreates the learner workspace
python3 scripts/project_test.py harness-bench --stage 1 --path /tmp/harness-bench-work --strictGrades one stage
python3 scripts/project_test.py harness-bench --all --solution --strictGrades the reference solution
go run . --budget 2 --out /tmp/orchard-budget.jsonReplays Orchard with 2 calls per policy
go run . --cases <cases.json> --endpoint <url> --model <model>Calls a live chat-completions model
--key-env <VAR>Names the environment variable that holds the key
Sources

Toy harness metrics in Python

resolved = (ftp_post == len(task.fail_to_pass)) and (ptp_broke == 0)
score = steps + modalities + toolsGAIA level: <=2 is 1, <=5 is 2, else 3
efficiency = steps / task.gold_stepstrajectory efficiency, 1.0 is gold
delta = r["accuracy_seen"] - r["accuracy_held"]
contam_flag = "*" if delta > 0.1 else " "held-set accuracy is canonical
vs_random = r["accuracy_held"] - rand_baselinebaseline 0.15
r = run_bench(sys, n_seen=40, n_held=160, seed=17)
Sources

CLI defaults

SettingDefaultNote
--casesfixtures/orchard-cases.jsonJSON array of cases
--recordingfixtures/orchard-recording.jsonIgnored when --endpoint is set
--budget4Model calls per policy, at most 10000
--timeout30sDeadline per policy, at most 5 minutes
--temperature0HTTP sampling temperature
--max-tokens128HTTP output token ceiling
Case limit1000Files are limited to one MiB
HTTP request capten secondsRedirects and query endpoints are rejected
Sources

Troubleshooting

SymptomCauseFix
Cases returns ErrConflictTwo cases share an IDGive each case a unique ID
Cases returns an error on a valid answerMisspelled Expected keyFix the key. Unknown fields are rejected
Leaderboard returns ErrConflictDataset digest, model digest or budget differRerun all policies on one config
15 min fails against 15 minutesExact answer contractDeclare variants in a separate contract
State is budget-exhaustedA retry spent the budgetLater cases stay in Total. Raise --budget
recording exhausted for requested promptPrompt missing or array used upAdd the exact policy prompt to responses
SWE-bench 50% drops to 35% on SWE-bench+Solution leakage in the issue textReport Verified and SWE-bench+ scores
Delta above 0.1, flag *Seen tasks score higher than held tasksQuote held-set accuracy only
Sources

Numbers to remember

SWE-bench+
32.67% of successful patches leaked solutions. 31.08% had weak tests.
Cutoffs
over 94% of SWE-bench issues predate most model cutoffs.
OSWorld-Human
top agents use 1.4-2.7x more steps than gold.
OSWorld-G
564 grounding-only samples. It splits grounding from planning.
Agent-team gain
~30-40 point delta on SWE-bench Pro.
Coordination tax
appears past ~4 agents on graph topology.
Orchard replay
baseline 1/3 in 3 calls, error-retry 2/3 in 4, evidence 3/3 in 3.
Not measured
cost, adversarial safety, your domain and the worst 1% of tails.
Sources

Claim checklist

Which benchmark and split?
Verified vs Pro is a large gap. A number on the wrong split is worthless.
Contamination check?
Was the benchmark released after the training cutoff? If not, treat with caution.
Which baseline?
Single-LLM, random and prior multi-agent work. Not an untuned copy of the same system.
Significance?
N trials, p-value and a confidence interval. Single runs mislead.
Cost disclosed?
Tokens per task and wall-clock. Report P50/P75/P95, not one number.
Comparable receipts?
Same dataset SHA-256, model config SHA-256 and call budget.
Retry on a wrong answer?
No. Error retries never inspect the expected answer.
Sources
Original source
---
id: agent-benchmarks
title: Agent Benchmarks: SWE-bench to OSWorld
code: AB
subtitle: Pick the benchmark, gate on the right tests, check contamination and compare runs on equal denominators.
hue: 28
sources: phases/14-agent-engineering/19-benchmarks-swebench-gaia, phases/14-agent-engineering/20-benchmarks-webarena-osworld, phases/16-multi-agent-and-swarms/24-evaluation-coordination-benchmarks, projects/harness-bench
source_hash: 7ff044c00ba88674
---

## Single-agent benchmarks side by side
type: comparison
span: 2
from: phases/14-agent-engineering/19-benchmarks-swebench-gaia/docs/en.md, phases/14-agent-engineering/20-benchmarks-webarena-osworld/docs/en.md

| Factor | SWE-bench | GAIA | WebArena | OSWorld |
|---|---|---|---|---|
| Tests | Code patching | Generalist tool use | Web tasks, long horizon | Real computer tasks |
| Size | 2,294 issues, 12 Python repos | 466 questions, 300 private | 812 tasks, 4 apps | 369 tasks, 3 OSes |
| Agent input | Codebase at pre-fix commit + issue | Question. Web, multi-modality | Self-hosted apps + utilities | 1920×1080 screenshots |
| Evaluator | FAIL_TO_PASS and PASS_TO_PASS tests | Not given | Execution-based, gym APIs | Not given |
| Best at release | SWE-agent 12.5% | GPT-4 with plugins 15% | GPT-4 agent 14.41% | Best model 12.24% |
| Human | Not given | 92% | 78.24% | 72.36% |
| Main trap | Contamination | Confusion with code benchmarks | Stale pinned app versions | GUI grounding |

## Multi-agent benchmarks side by side
type: comparison
span: 2
from: phases/16-multi-agent-and-swarms/24-evaluation-coordination-benchmarks/docs/en.md

| Factor | MARBLE | COMMA | MedAgentBoard | SWE-bench Pro |
|---|---|---|---|---|
| Measures | Star, chain, tree, graph topologies | Multimodal asymmetric information | 4 medical task categories | Uncontaminated, later cutoffs |
| Scoring | Milestone KPIs, partial progress | Agent-agent collaboration | Multi-agent vs single-LLM vs rules | 1865 problems, 41 repos |
| Headline | Graph best for research | Frontier models struggle vs random | Multi-agent does NOT dominate | ~23% on Pro vs 70%+ on Verified |
| Lesson | Cognitive planning adds ~3% | Measure before you claim | Many systems are over-engineered | Verified alone is not evidence |

## Use X when
type: tips
from: phases/14-agent-engineering/19-benchmarks-swebench-gaia/docs/en.md, phases/14-agent-engineering/20-benchmarks-webarena-osworld/docs/en.md, phases/16-multi-agent-and-swarms/24-evaluation-coordination-benchmarks/docs/en.md

- SWE-bench Verified: a code agent. Always report Verified scores.
- SWE-bench Pro: a contamination reality check. Pro is the current gating test.
- GAIA: a generalist agent. Use the private leaderboard split.
- AgentBench: a multi-environment comparison across 8 environments.
- WebArena Verified: a web agent. Self-host it for continuous evaluation.
- OSWorld: a desktop agent. Run it in a VM fleet.
- MARBLE: you compare coordination topologies apples-to-apples.
- AgentArch: you must justify each layer of an enterprise stack.
- Custom evals: your product flows. Capture gold trajectories for your top 20 tasks.

## SWE-bench gate: run_task
type: flow
from: phases/14-agent-engineering/19-benchmarks-swebench-gaia/code/main.py, phases/14-agent-engineering/19-benchmarks-swebench-gaia/docs/en.md

1. **Harness**: copy `state_before`. Count passing FAIL_TO_PASS and PASS_TO_PASS checks.
2. **Agent**: apply the patch to a copy of the state.
3. **Harness**: count passing checks again on the new state.
4. **Fix gate**: all FAIL_TO_PASS tests must pass after the patch.
5. **No-regression gate**: `ptp_broke` must be 0.
6. **Result**: `resolved` is true only when both gates pass.
7. **Report**: resolution rate is resolved tasks over all tasks.

## Harness Bench pipeline
type: architecture
from: projects/harness-bench/solution, projects/harness-bench/README.md

```arch
node | cases | 1,1 | Cases JSON | ID, Prompt, Expected | store
node | load | 2,1 | Cases() | strict decode, max 1000
node | eval | 3,1 | EvaluatePolicy() | 3 policies, call budget | process
node | board | 4,1 | Leaderboard() | equal denominators
node | rec | 2,2 | Recording | authored_fixture | store
node | http | 3,2 | HTTPModel() | chat-completions | model
node | score | 4,2 | Correct() | exact answer contract
edge | cases | load |
edge | load | eval | cases
edge | rec | eval | NewModel
edge | http | eval | ContextModel
edge | eval | score | answer
edge | eval | board | Result
```

## Run the lessons and the project
type: command-table
span: 2
from: phases/14-agent-engineering/19-benchmarks-swebench-gaia/docs/en.md, phases/14-agent-engineering/20-benchmarks-webarena-osworld/docs/en.md, phases/16-multi-agent-and-swarms/24-evaluation-coordination-benchmarks/docs/en.md, projects/harness-bench/README.md, projects/harness-bench/stages

| Command | What it does |
|---|---|
| `python3 code/main.py` | Runs the toy harness of lessons 14.19, 14.20 and 16.24 |
| `python3 scripts/project_test.py harness-bench --init /tmp/harness-bench-work` | Creates the learner workspace |
| `python3 scripts/project_test.py harness-bench --stage 1 --path /tmp/harness-bench-work --strict` | Grades one stage |
| `python3 scripts/project_test.py harness-bench --all --solution --strict` | Grades the reference solution |
| `go run . --budget 2 --out /tmp/orchard-budget.json` | Replays Orchard with 2 calls per policy |
| `go run . --cases <cases.json> --endpoint <url> --model <model>` | Calls a live chat-completions model |
| `--key-env <VAR>` | Names the environment variable that holds the key |

## Toy harness metrics in Python
type: annotated-code
lang: python
from: phases/14-agent-engineering/19-benchmarks-swebench-gaia/code/main.py, phases/14-agent-engineering/20-benchmarks-webarena-osworld/code/main.py, phases/16-multi-agent-and-swarms/24-evaluation-coordination-benchmarks/code/main.py

```python
resolved = (ftp_post == len(task.fail_to_pass)) and (ptp_broke == 0)
score = steps + modalities + tools    # GAIA level: <=2 is 1, <=5 is 2, else 3
efficiency = steps / task.gold_steps  # trajectory efficiency, 1.0 is gold
delta = r["accuracy_seen"] - r["accuracy_held"]
contam_flag = "*" if delta > 0.1 else " "  # held-set accuracy is canonical
vs_random = r["accuracy_held"] - rand_baseline  # baseline 0.15
r = run_bench(sys, n_seen=40, n_held=160, seed=17)
```

## CLI defaults
type: defaults
from: projects/harness-bench/solution/main.go, projects/harness-bench/README.md

| Setting | Default | Note |
|---|---|---|
| `--cases` | fixtures/orchard-cases.json | JSON array of cases |
| `--recording` | fixtures/orchard-recording.json | Ignored when `--endpoint` is set |
| `--budget` | 4 | Model calls per policy, at most 10000 |
| `--timeout` | 30s | Deadline per policy, at most 5 minutes |
| `--temperature` | 0 | HTTP sampling temperature |
| `--max-tokens` | 128 | HTTP output token ceiling |
| Case limit | 1000 | Files are limited to one MiB |
| HTTP request cap | ten seconds | Redirects and query endpoints are rejected |

## Troubleshooting
type: troubleshooting
span: 2
from: projects/harness-bench/stages, projects/harness-bench/solution, projects/harness-bench/README.md, phases/14-agent-engineering/19-benchmarks-swebench-gaia/docs/en.md, phases/14-agent-engineering/20-benchmarks-webarena-osworld/docs/en.md, phases/16-multi-agent-and-swarms/24-evaluation-coordination-benchmarks/code/main.py

| Symptom | Cause | Fix |
|---|---|---|
| `Cases` returns `ErrConflict` | Two cases share an ID | Give each case a unique ID |
| `Cases` returns an error on a valid answer | Misspelled `Expected` key | Fix the key. Unknown fields are rejected |
| `Leaderboard` returns `ErrConflict` | Dataset digest, model digest or budget differ | Rerun all policies on one config |
| `15 min` fails against `15 minutes` | Exact answer contract | Declare variants in a separate contract |
| State is `budget-exhausted` | A retry spent the budget | Later cases stay in `Total`. Raise `--budget` |
| `recording exhausted for requested prompt` | Prompt missing or array used up | Add the exact policy prompt to `responses` |
| SWE-bench 50% drops to 35% on SWE-bench+ | Solution leakage in the issue text | Report Verified and SWE-bench+ scores |
| Delta above 0.1, flag `*` | Seen tasks score higher than held tasks | Quote held-set accuracy only |

## Numbers to remember
type: concepts
from: phases/14-agent-engineering/19-benchmarks-swebench-gaia/docs/en.md, phases/14-agent-engineering/20-benchmarks-webarena-osworld/docs/en.md, phases/16-multi-agent-and-swarms/24-evaluation-coordination-benchmarks/docs/en.md, projects/harness-bench/README.md

- **SWE-bench+**: 32.67% of successful patches leaked solutions. 31.08% had weak tests.
- **Cutoffs**: over 94% of SWE-bench issues predate most model cutoffs.
- **OSWorld-Human**: top agents use 1.4-2.7x more steps than gold.
- **OSWorld-G**: 564 grounding-only samples. It splits grounding from planning.
- **Agent-team gain**: ~30-40 point delta on SWE-bench Pro.
- **Coordination tax**: appears past ~4 agents on graph topology.
- **Orchard replay**: baseline 1/3 in 3 calls, error-retry 2/3 in 4, evidence 3/3 in 3.
- **Not measured**: cost, adversarial safety, your domain and the worst 1% of tails.

## Claim checklist
type: qa
from: phases/16-multi-agent-and-swarms/24-evaluation-coordination-benchmarks/docs/en.md, phases/14-agent-engineering/19-benchmarks-swebench-gaia/docs/en.md, projects/harness-bench/README.md

- **Which benchmark and split?** Verified vs Pro is a large gap. A number on the wrong split is worthless.
- **Contamination check?** Was the benchmark released after the training cutoff? If not, treat with caution.
- **Which baseline?** Single-LLM, random and prior multi-agent work. Not an untuned copy of the same system.
- **Significance?** N trials, p-value and a confidence interval. Single runs mislead.
- **Cost disclosed?** Tokens per task and wall-clock. Report P50/P75/P95, not one number.
- **Comparable receipts?** Same dataset SHA-256, model config SHA-256 and call budget.
- **Retry on a wrong answer?** No. Error retries never inspect the expected answer.
Code and artifacts 1
What you take away

Saved in your notebook on this device.

Sources and limits

Reading checkpoints are self-reported. Test results are recorded separately and apply to the executed cases.

AI Engineering from Scratch · 3f9715c · MIT

Download this file ↓