EP16 and EP32 at two batch sizes
A two by two comparison of DeepSeek-R1 decode throughput on H20 from Ant Group. At batch 8, EP16 gave 278 tokens per second and EP32 gave 293. At batch 32, EP16 gave 675 and EP32 gave 585. About half of the MoE traffic stayed on NVLink at EP16 and about a quarter at EP32.
DEEPSEEK-R1 DECODE ON H20
--ep-size 16
--ep-size 32
batch 8
278 tokens/s
slower at this batch
293 tokens/s
faster at this batch
batch 32
675 tokens/s
faster at this batch
585 tokens/s
slower at this batch
MoE traffic
about 50% on NVLink
about 25% on NVLink
Measured by Ant Group, LMSYS blog, 2025-09-26.