Computer use — Claude, OpenAI CUA, Gemini 2.5 (all vision-based)
Claude computer use
Anthropic, Oct 22 2024
screenshot in keyboard/mouse out
no accessibility APIs; pure pixels
computer tool baked into model
schema not developer-configurable
trained to count pixels
from reference points to targets
resolution-independent coordinates
OpenAI CUA / Operator
Jan 2025 ChatGPT agent mode Jul 2025
GPT-4o variant with RL on GUI
Responses API, computer-use-preview
OSWorld 38.1%, WebArena 58.1%
WebVoyager 87% at launch
consumer-facing launch path
merged into ChatGPT agent mode
mobile + web first
Gemini 2.5 Computer Use
Google DeepMind, Oct 7 2025
browser-only (13 actions)
lowest latency of the three
Online-Mind2Web ~70%
Gemini 3 Flash: built-in
per-step safety service
every action assessed before execution
rejects unsafe without model involvement
shared 2026 contract: all inputs except the user message are untrusted
1. per-step safety classifier before every action
2. allowlist / blocklist on navigation targets
3. human-in-the-loop on sensitive actions (login, purchase, CAPTCHA)
4. content capture to external store; span references only on traces
5. refusal on retrieved-text directives ("ignore all instructions")
"only direct instructions from the user count as permission" — OpenAI CUA docs
Greshake et al. (2023) is the threat model; all three productions take it seriously.