Multimodal agent loop — perceive, reason, act, observe
the four-phase loop
1. perceive
screenshot
+ accessibility tree
+ goal + history
VLM encoding
high-res 1280+
native aspect
2. reason
plan next step
ground instruction
to coordinates
emit JSON action
{action, x, y, desc}
element_desc helps
3. act
dispatch to browser
Selenium / CDP
or OS driver
verify success
element exists?
retry on failure
4. observe
new screenshot
state diff
goal check
repeat or done
max steps ~30
memory + benchmarks
memory compression
1. summary-chain: summary every 5 steps
2. skip-frame: first + last + every 3rd
3. log-only: text log of actions (Claude)
4. hybrid: log + last-2 screenshots + summary
30 screenshots at 2048 px = 360k tokens
compression is mandatory past step ~10
2026 benchmarks
ScreenSpot-Pro : Claude 4.7 ~90 / Qwen2.5-VL 85
VisualWebArena : Gemini 3 Pro 27 / open ~20
AgentVista : frontier 27-40 / open 10-20
Ferret-UI : GPT-5 ~82 / Qwen2.5-VL ~70
still hard
long-horizon + recovery + fine grounding