CTDE and the three MARL baselines
Training time — centralized
critic / mixer / value
sees all agents' observations + actions
stationary target for each actor update
addresses non-stationarity of multi-learner settings
MADDPG critic · QMIX mixer · MAPPO centralized V
Deploy time — decentralized
actor 0
sees o_0 only
local action
actor 1
sees o_1 only
local action
actor N
sees o_N only
local action
no critic at runtime; only local policy call
three baselines, three specializations
MADDPG (2017)
per-agent critic · continuous actions · mixed cooperative/competitive · scales ~10 agents
QMIX (2018)
value decomposition · monotone mixer · cooperative · discrete · dominant on SMAC
MAPPO (2022)
PPO with centralized V · minimal tuning · cooperative · robust across benchmarks
production note
start with MAPPO; escalate to MADDPG for mixed settings, QMIX for sparse cooperative
For LLM-agent systems: CTDE is a design pattern; training end-to-end is optional.