One token through an MoE layer
Four stages from left to right. TopK picks experts for each token. The dispatcher, chosen with moe-a2a-backend, moves tokens to the GPUs that hold those experts. MoeRunner computes the local experts with a grouped matrix multiply chosen with moe-runner-backend. The dispatcher combines the results and moves them back.
ONE MOE LAYER, EVERY STEP
TopK
picks experts
for each token
dispatch
tokens move to
expert GPUs
MoeRunner
grouped matrix
multiply
combine
results move
back to tokens
--moe-a2a-backend
--moe-runner-backend
--moe-a2a-backend
--deepep-mode auto: normal for prefill batches, low_latency for decode
all-to-all backends set --ep-size to --tp-size