Swarm Optimization and MARL for Agents · AI Engineering

PSO and ACO for prompts and routing, queue swarms, and MADDPG, QMIX, MAPPO and self-play set side by side.

PSO vs ACO

Factor	PSO	ACO
Problem	Continuous search space	Routing or path selection
Memory	`p_best` and `g_best`	Pheromone per task-type × agent
Agent use	LMPSO, Model Swarms	AMRO-S routing
Reported gain	13.3% over 12 baselines	4.7x speedup
Population	10-30 particles	3 agents, 4 task types in the demo
Update rule	Inertia + cognitive + social	Decay, then gated deposit
Demo function	`run_lmpso()`	`run_amro_s()`

Sources

MARL algorithms

Factor	MADDPG	QMIX	MAPPO
Year, venue	NeurIPS 2017	ICML 2018	NeurIPS 2022
Setting	Coop, competitive, mixed	Cooperative only	Cooperative
Training signal	Per-agent critic sees all actions	Monotone mixer of per-agent Qs	Centralized value function
Policy type	Off-policy	Off-policy	On-policy PPO
Testbed	Particle World	SMAC	Football, Hanabi, MPE
Limit	Past ~10 agents critics grow	Reward must be monotone	Default baseline to beat
Demo	`run_maddpg_style`	`run_qmix_style`	`run_mappo_style`

Sources

MARL regimes

Factor	IQL / IPPO	CTDE	Self-play	League play
Idea	Others are environment	Central critic, local policy	Copy of past self	Population of past policies
Fits	Loosely-coupled tasks	Cooperative navigation	Zero-sum games	Strategy cycles
Examples	Independent Q	MADDPG, COMA, MAPPO, QMIX	AlphaZero	AlphaStar
Note	Theoretical convergence: none	Most common modern paradigm	Can stick in a dominance cycle	Adds exploiters and main exploiters

Sources

phases/09-reinforcement-learning/10-multi-agent-rl/docs/en.md

Queue swarm vs fixed assignment

Factor	Sequential	Fixed assignment	Swarm
Who picks work	One worker	Pre-assigned worker id	Workers pull from a shared queue
Function	`run_sequential`	`run_fixed_assignment`	`run_swarm`
Load balance	None	Fast workers sit idle	Automatic
Fits	Baseline only	Durations known ahead	Many independent tasks
Weak point	Wall-clock time	Worker 0 gets all slow tasks	Ordering and debugging

Sources

Use each method when

PSO: continuous parameters, cheap automatic fitness, population 10-30.
ACO: routing that repeats over time and needs interpretable evidence.
Neither: fitness needs human review, or latency must be strict.
Genetic algorithm: discrete combinatorial space that PSO does not cover.
Swarm: many independent tasks with variable duration.
Supervisor: the task needs one coherent global plan.
MAPPO: start here for cooperative MARL.
Self-play: zero-sum games. League play when strategies cycle.

Sources

Run the patterns

Task	Code
Run any lesson demo	`python3 code/main.py`
PSO over 2 prompt parameters	`history = run_lmpso()`
ACO routing vs random	`run_amro_s(n_tasks=200, seed=0)`
Print the trail	`print_pheromone_table(router)`
Steps-to-goal per MARL style	`bench(label, runner)` over 500 trials
Pull from a shared queue	`run_swarm(tasks, n_workers)`
Independent Q on the grid	`independent_q(episodes=1500)`
Joint Q over 16 actions	`joint_q_learning(episodes=1500)`

Sources

PSO update

w, c1, c2 = 0.6, 1.2, 1.2inertia, cognitive, social

r1, r2 = rng.random(), rng.random()

cognitive = c1 * r1 * (p.p_best[d] - p.x[d])

social = c2 * r2 * (g_best_x[d] - p.x[d])

p.v[d] = w * p.v[d] + cognitive + social

p.x[d] += p.v[d]

p.x[d] = max(bounds[d][0], min(bounds[d][1], p.x[d]))clamp to bounds

f = fitness(p.x)only an evaluator, no gradient

if f > p.p_best_fit:then check g_best_fit

Sources

phases/16-multi-agent-and-swarms/19-swarm-optimization-pso-aco/code/main.py

ACO router

router = PheromoneRouter(task_types, agents)every trail starts at 1.0

aco_agent = router.choose(tt, rng)sample by pheromone weight

aq = simulate_task(aco_agent, tt, rng)

router.deposit(tt, aco_agent, aq)

self.pheromones[task_type][a] *= (1.0 - self.decay)all agents decay

if quality >= self.quality_threshold:quality gate

self.pheromones[task_type][agent] += self.reinforce * quality

Sources

phases/16-multi-agent-and-swarms/19-swarm-optimization-pso-aco/code/main.py

CTDE pattern

Nodes and connections

Node	Role	Kind
Global state	all obs + joint action	store
Centralized critic	training time only	model
Agent 1 obs	o_i
Actor 1	mu_i(o_i)	process
Agent 2 obs	o_i
Actor 2	mu_i(o_i)	process

From	To	Connection
Global state	Centralized critic	train
Centralized critic	Actor 1	gradient
Agent 1 obs	Actor 1	deploy
Agent 2 obs	Actor 2	deploy
Centralized critic	Actor 2	gradient

Sources

Defaults in the lesson code

Setting	Default	Note
`n_particles`	20	`run_lmpso` population
`iterations`	30	PSO iterations
`decay`	0.05	Pheromone decay per deposit
`reinforce`	0.2	Deposit is reinforce times quality
`quality_threshold`	0.6	No deposit below it
`n_tasks`	200	Routed tasks in `run_amro_s`
`epsilon`	0.15	Exploration in `independent_q`
`gamma`	0.95	Discount, with `alpha` 0.1

Sources

Failure modes

Symptom	Cause	Fix
Router locks on a bad agent	Fast-but-wrong agent got pheromone	Quality-gate the deposits
Best fitness diverges	Catastrophic drift after data shift	Reset or double the decay rate
Swarm forgets good solutions	Decay too fast	Slower decay, monitor best fitness
Long tasks never run	Queue starvation	Priority queue with aging
Task output appears twice	Worker crashed mid-run	Idempotent workers, durable queue
Agents chase each other	Policy drift, non-stationarity	Centralized critic, slow learning rates
Old replay hurts learning	Transitions from obsolete opponents	Relabel or weight by recency
Auction agents bid zero	Reward hacking via coordination	Careful reward design, constraints

Sources

Questions

Why gradient-free for prompts?: A prompt is a discrete string. PSO and ACO need only an evaluator.
What does a 20-particle, 50-iteration PSO cost?: About 20 at ~0.02 per call.
Why does QMIX decentralize?: Monotone mixing lets each agent take its own argmax.
Why not train MARL for LLM agents?: Most systems prompt policies. Train only with data, reward and infrastructure.
Why is the joint-Q demo slow to scale?: Its action space is |A|² = 16 for two agents.

Sources

Ship checklist

Start with 10-20 particles and 20-50 iterations.
Log pheromones or g_best per iteration.
Emit a cost-per-iteration metric.
Give every swarm task a trace ID. Apply back-pressure when the queue grows.
Let the execution path see only o_i.
Log every agent's observation and action stream.

Sources

Original source

---
id: swarm-optimization
title: Swarm Optimization and MARL for Agents
code: SM
subtitle: PSO and ACO for prompts and routing, queue swarms, and MADDPG, QMIX, MAPPO and self-play set side by side.
hue: 160
sources: phases/16-multi-agent-and-swarms/19-swarm-optimization-pso-aco, phases/16-multi-agent-and-swarms/20-marl-maddpg-qmix-mappo, phases/16-multi-agent-and-swarms/09-parallel-swarm-networks, phases/09-reinforcement-learning/10-multi-agent-rl
source_hash: 9890d657357bead1
---

## PSO vs ACO
type: comparison
from: phases/16-multi-agent-and-swarms/19-swarm-optimization-pso-aco/docs/en.md, phases/16-multi-agent-and-swarms/19-swarm-optimization-pso-aco/code/main.py

| Factor | PSO | ACO |
|---|---|---|
| Problem | Continuous search space | Routing or path selection |
| Memory | `p_best` and `g_best` | Pheromone per task-type × agent |
| Agent use | LMPSO, Model Swarms | AMRO-S routing |
| Reported gain | 13.3% over 12 baselines | 4.7x speedup |
| Population | 10-30 particles | 3 agents, 4 task types in the demo |
| Update rule | Inertia + cognitive + social | Decay, then gated deposit |
| Demo function | `run_lmpso()` | `run_amro_s()` |

## MARL algorithms
type: comparison
from: phases/16-multi-agent-and-swarms/20-marl-maddpg-qmix-mappo/docs/en.md, phases/16-multi-agent-and-swarms/20-marl-maddpg-qmix-mappo/code/main.py

| Factor | MADDPG | QMIX | MAPPO |
|---|---|---|---|
| Year, venue | NeurIPS 2017 | ICML 2018 | NeurIPS 2022 |
| Setting | Coop, competitive, mixed | Cooperative only | Cooperative |
| Training signal | Per-agent critic sees all actions | Monotone mixer of per-agent Qs | Centralized value function |
| Policy type | Off-policy | Off-policy | On-policy PPO |
| Testbed | Particle World | SMAC | Football, Hanabi, MPE |
| Limit | Past ~10 agents critics grow | Reward must be monotone | Default baseline to beat |
| Demo | `run_maddpg_style` | `run_qmix_style` | `run_mappo_style` |

## MARL regimes
type: comparison
from: phases/09-reinforcement-learning/10-multi-agent-rl/docs/en.md

| Factor | IQL / IPPO | CTDE | Self-play | League play |
|---|---|---|---|---|
| Idea | Others are environment | Central critic, local policy | Copy of past self | Population of past policies |
| Fits | Loosely-coupled tasks | Cooperative navigation | Zero-sum games | Strategy cycles |
| Examples | Independent Q | MADDPG, COMA, MAPPO, QMIX | AlphaZero | AlphaStar |
| Note | Theoretical convergence: none | Most common modern paradigm | Can stick in a dominance cycle | Adds exploiters and main exploiters |

## Queue swarm vs fixed assignment
type: comparison
from: phases/16-multi-agent-and-swarms/09-parallel-swarm-networks/docs/en.md, phases/16-multi-agent-and-swarms/09-parallel-swarm-networks/code/main.py

| Factor | Sequential | Fixed assignment | Swarm |
|---|---|---|---|
| Who picks work | One worker | Pre-assigned worker id | Workers pull from a shared queue |
| Function | `run_sequential` | `run_fixed_assignment` | `run_swarm` |
| Load balance | None | Fast workers sit idle | Automatic |
| Fits | Baseline only | Durations known ahead | Many independent tasks |
| Weak point | Wall-clock time | Worker 0 gets all slow tasks | Ordering and debugging |

## Use each method when
type: concepts
from: phases/16-multi-agent-and-swarms/19-swarm-optimization-pso-aco/docs/en.md, phases/16-multi-agent-and-swarms/20-marl-maddpg-qmix-mappo/docs/en.md, phases/16-multi-agent-and-swarms/09-parallel-swarm-networks/docs/en.md, phases/09-reinforcement-learning/10-multi-agent-rl/docs/en.md

- **PSO**: continuous parameters, cheap automatic fitness, population 10-30.
- **ACO**: routing that repeats over time and needs interpretable evidence.
- **Neither**: fitness needs human review, or latency must be strict.
- **Genetic algorithm**: discrete combinatorial space that PSO does not cover.
- **Swarm**: many independent tasks with variable duration.
- **Supervisor**: the task needs one coherent global plan.
- **MAPPO**: start here for cooperative MARL.
- **Self-play**: zero-sum games. League play when strategies cycle.

## Run the patterns
type: task-table
from: phases/16-multi-agent-and-swarms/19-swarm-optimization-pso-aco/code/main.py, phases/16-multi-agent-and-swarms/20-marl-maddpg-qmix-mappo/code/main.py, phases/16-multi-agent-and-swarms/09-parallel-swarm-networks/code/main.py, phases/09-reinforcement-learning/10-multi-agent-rl/code/main.py, phases/16-multi-agent-and-swarms/19-swarm-optimization-pso-aco/docs/en.md

| Task | Code |
|---|---|
| Run any lesson demo | `python3 code/main.py` |
| PSO over 2 prompt parameters | `history = run_lmpso()` |
| ACO routing vs random | `run_amro_s(n_tasks=200, seed=0)` |
| Print the trail | `print_pheromone_table(router)` |
| Steps-to-goal per MARL style | `bench(label, runner)` over 500 trials |
| Pull from a shared queue | `run_swarm(tasks, n_workers)` |
| Independent Q on the grid | `independent_q(episodes=1500)` |
| Joint Q over 16 actions | `joint_q_learning(episodes=1500)` |

## PSO update
type: annotated-code
lang: python
from: phases/16-multi-agent-and-swarms/19-swarm-optimization-pso-aco/code/main.py

```python
w, c1, c2 = 0.6, 1.2, 1.2                  # inertia, cognitive, social
r1, r2 = rng.random(), rng.random()
cognitive = c1 * r1 * (p.p_best[d] - p.x[d])
social = c2 * r2 * (g_best_x[d] - p.x[d])
p.v[d] = w * p.v[d] + cognitive + social
p.x[d] += p.v[d]
p.x[d] = max(bounds[d][0], min(bounds[d][1], p.x[d]))  # clamp to bounds
f = fitness(p.x)                           # only an evaluator, no gradient
if f > p.p_best_fit:                       # then check g_best_fit
```

## ACO router
type: annotated-code
lang: python
from: phases/16-multi-agent-and-swarms/19-swarm-optimization-pso-aco/code/main.py

```python
router = PheromoneRouter(task_types, agents)   # every trail starts at 1.0
aco_agent = router.choose(tt, rng)             # sample by pheromone weight
aq = simulate_task(aco_agent, tt, rng)
router.deposit(tt, aco_agent, aq)
self.pheromones[task_type][a] *= (1.0 - self.decay)    # all agents decay
if quality >= self.quality_threshold:                   # quality gate
    self.pheromones[task_type][agent] += self.reinforce * quality
```

## CTDE pattern
type: architecture
from: phases/16-multi-agent-and-swarms/20-marl-maddpg-qmix-mappo/docs/en.md, phases/09-reinforcement-learning/10-multi-agent-rl/docs/en.md

```arch
node | state | 1,1 | Global state | all obs + joint action | store
node | critic | 2,1 | Centralized critic | training time only | model
node | o1 | 1,2 | Agent 1 obs | o_i |
node | a1 | 2,2 | Actor 1 | mu_i(o_i) | process
node | o2 | 3,2 | Agent 2 obs | o_i |
node | a2 | 4,2 | Actor 2 | mu_i(o_i) | process
edge | state | critic | train
edge | critic | a1 | gradient
edge | o1 | a1 | deploy
edge | o2 | a2 | deploy
edge | critic | a2 | gradient
```

## Defaults in the lesson code
type: defaults
from: phases/16-multi-agent-and-swarms/19-swarm-optimization-pso-aco/code/main.py, phases/09-reinforcement-learning/10-multi-agent-rl/code/main.py

| Setting | Default | Note |
|---|---|---|
| `n_particles` | 20 | `run_lmpso` population |
| `iterations` | 30 | PSO iterations |
| `decay` | 0.05 | Pheromone decay per deposit |
| `reinforce` | 0.2 | Deposit is reinforce times quality |
| `quality_threshold` | 0.6 | No deposit below it |
| `n_tasks` | 200 | Routed tasks in `run_amro_s` |
| `epsilon` | 0.15 | Exploration in `independent_q` |
| `gamma` | 0.95 | Discount, with `alpha` 0.1 |

## Failure modes
type: troubleshooting
from: phases/16-multi-agent-and-swarms/19-swarm-optimization-pso-aco/docs/en.md, phases/16-multi-agent-and-swarms/09-parallel-swarm-networks/docs/en.md, phases/09-reinforcement-learning/10-multi-agent-rl/docs/en.md, phases/16-multi-agent-and-swarms/20-marl-maddpg-qmix-mappo/docs/en.md

| Symptom | Cause | Fix |
|---|---|---|
| Router locks on a bad agent | Fast-but-wrong agent got pheromone | Quality-gate the deposits |
| Best fitness diverges | Catastrophic drift after data shift | Reset or double the decay rate |
| Swarm forgets good solutions | Decay too fast | Slower decay, monitor best fitness |
| Long tasks never run | Queue starvation | Priority queue with aging |
| Task output appears twice | Worker crashed mid-run | Idempotent workers, durable queue |
| Agents chase each other | Policy drift, non-stationarity | Centralized critic, slow learning rates |
| Old replay hurts learning | Transitions from obsolete opponents | Relabel or weight by recency |
| Auction agents bid zero | Reward hacking via coordination | Careful reward design, constraints |

## Questions
type: qa
from: phases/16-multi-agent-and-swarms/19-swarm-optimization-pso-aco/docs/en.md, phases/16-multi-agent-and-swarms/20-marl-maddpg-qmix-mappo/docs/en.md, phases/09-reinforcement-learning/10-multi-agent-rl/docs/en.md

- **Why gradient-free for prompts?** A prompt is a discrete string. PSO and ACO need only an evaluator.
- **What does a 20-particle, 50-iteration PSO cost?** About $20 at ~$0.02 per call.
- **Why does QMIX decentralize?** Monotone mixing lets each agent take its own argmax.
- **Why not train MARL for LLM agents?** Most systems prompt policies. Train only with data, reward and infrastructure.
- **Why is the joint-Q demo slow to scale?** Its action space is `|A|² = 16` for two agents.

## Ship checklist
type: tips
from: phases/16-multi-agent-and-swarms/19-swarm-optimization-pso-aco/docs/en.md, phases/16-multi-agent-and-swarms/20-marl-maddpg-qmix-mappo/docs/en.md, phases/16-multi-agent-and-swarms/09-parallel-swarm-networks/docs/en.md

- Start with 10-20 particles and 20-50 iterations.
- Log pheromones or g_best per iteration.
- Emit a cost-per-iteration metric.
- Give every swarm task a trace ID. Apply back-pressure when the queue grows.
- Let the execution path see only `o_i`.
- Log every agent's observation and action stream.