WebArena and OSWorld — web and desktop agent benchmarks
WebArena web agents
812 tasks, 4 self-hosted apps
shopping, forum, gitlab-like, business CMS
+ utilities: map, calculator, scratchpad
execution-based evaluation via gym APIs
release: GPT-4 agent 14.41%, human 78.24%
extensions: VisualWebArena, TheAgentCompany
pinned app versions reproducible scoring
OSWorld desktop agents
369 tasks, Ubuntu + Windows + macOS
1920x1080 screenshots as observation
keyboard + mouse control of real apps
release: best model 12.24%, human 72.36%
OSWorld-G: 564-sample grounding suite + Jedi training
OSWorld-Human: expert gold trajectories
top agents use 1.4-2.7x more steps than needed
two primary failure modes
1. GUI grounding pixel to element mapping
model cannot localize a button reliably; mis-clicks dominate low-level failures
2. operational knowledge which menu, which shortcut
tail knowledge humans build over years; agents guess from training snapshots
production fix: Claude/OpenAI CUA/Gemini computer use (Lesson 21)
vision-based; trained on WebArena/OSWorld-shaped workloads;
per-step safety services assess each action before execution
report success rate AND trajectory efficiency (steps / gold) on every run.