Two data parallel copies of one model
A data parallel controller receives requests and gives them to two model worlds in turn, because load-balance-method auto resolves to round_robin outside prefill-decode disaggregation. World 0 uses GPUs g0 to g3 and world 1 uses g4 to g7, and each world has its own scheduler and its own KV cache.
data parallel controller
--load-balance-method auto: round_robin
world 0 · --tp-size 4
g0
g1
g2
g3
its own KV cache
the other world never sees it
world 1 · --tp-size 4
g4
g5
g6
g7
its own KV cache
the other world never sees it
requests 1, 3, 5
requests 2, 4, 6