One TP group of eight GPUs, split three ways
Eight GPUs g0 to g7 drawn in three rows. With tp-size 8 every layer is split across all eight. With attn-dp-size 4 the attention layers run in four groups of two GPUs, g0 and g1, g2 and g3, g4 and g5, g6 and g7, each with its own scheduler and KV cache. With ep-size 8 the expert layers use the same eight GPUs as one group.
FLAG AND LAYER
THE SAME EIGHT GPUS
--tp-size 8
dense and FFN layers
g0
g1
g2
g3
g4
g5
g6
g7
one TP group: each weight matrix is cut in 8
--attn-dp-size 4
attention layers
group 0
group 1
group 2
group 3
g0
g1
g2
g3
g4
g5
g6
g7
each group has its own scheduler, requests, and KV cache
--ep-size 8
expert layers
g0
g1
g2
g3
g4
g5
g6
g7
one expert group, equal to the TP group