Theory of Mind orders and the coordination illusion
zeroth-order
no model of others
agent acts on own observations only
collides with other agents
duplicates effort on shared resources
measured: ~0.96 collisions / trial
(3 agents, 3 boxes, no ToM)
first-order
"A believes X"
agent tracks each other's beliefs
predicts action from modeled beliefs
avoids redundancy + anticipates need
measured: ~0.00 collisions / trial
(with preference prime)
second-order
"A believes B believes X"
recursive belief modeling
emerges in cooperative games
degrades on long horizons
Li et al. 2023 context failure mode:
identity-belief binding drops over turns
the coordination illusion (Riedl 2025, arXiv:2510.05174)
claim
"our agents coordinate emergently"
failure
coordination is prompt-dressed; strip "work together" from system prompt, effect vanishes
audit
require control condition, statistical test, complementarity metric, capacity-check
real
identity-linked differentiation + goal-directed complementarity + higher-order synergy
only
the ToM-prompted condition produces all three above chance in Riedl's population experiment
larger models show some coordination without ToM prompting, but the effect is smaller