Swarm optimization for LLMs — PSO and ACO applied to prompts and routing
PSO — Particle Swarm Optimization
parameter space (2D here; in practice: prompt embedding, LoRA delta)
optimum
v = w*v + c1*r1*(p_best - x) + c2*r2*(g_best - x)
LMPSO: velocity is a prompt; Model Swarms: particles are expert weights (+13.3%)
ACO — Ant Colony Optimization
task_type coder mathematician writer
code 2.96 0.08 0.08
math 0.08 3.21 0.08
writing 0.08 0.08 2.97
planning 0.08 0.25 0.88
pheromone(t, a) *= (1 - decay); += reinforce*quality on quality-gated deposit
AMRO-S: 4.7x speedup; interpretable routing evidence; async update
when to use what
PSO
continuous numeric or low-dim continuous embedding · cheap fitness · pop 10-30
ACO
routing / path selection · decisions reinforce over time · need interpretable evidence
GA
discrete symbolic / program synthesis · population-based, crossover-driven
DPO/RL
differentiable reward · large budget · training-time optimization
manual
budget < $100 · single-digit parameters · cache hits worth more than search
all swarm optimizers need a quality gate to avoid locking in on fast-but-wrong candidates