four MARL regimes, same environment shape
independent PPO / Q
each agent has its own
policy and critic
treats others as env
simple, no coord signal
fails on tight coupling
CTDE (MAPPO/QMIX)
decentralized actor:
pi_i(a_i | o_i)
centralized critic:
Q(s, a_1, .., a_n)
2026 cooperative default
self-play
single agent, two roles
opponent = past snapshot
symmetric gradients
zero-sum games
AlphaGo / AlphaZero
league play
population of
past + current
policies; sample
opponents
AlphaStar
the core difficulty
from agent i's view, the environment now includes pi_{-i}
as pi_{-i} learns, the transition P(s' | s, a_i) is non-stationary
Markov property breaks -> tabular convergence proofs no longer apply
CTDE fixes this by conditioning the critic on the full joint action; self-play fixes it by structure
add communication, language, or MCTS and you get StarCraft II / Dota 2 / multi-agent LLM systems