DualPipe schedule — P=4, 8 micro-batches, two directions
rank 0
rank 1
rank 2
rank 3
time →
F1
F2
F3
F4
F5/F8R
F6/F7R
F7/F6R
F8/F5R
B1
B2
B3
B4
F1
F2
F3
F4/F8R
F5/F7R
F6/F6R
F7/F5R
F8/B1
B2
B3
B4
F1
F2
F3/F8R
F4/F7R
F5/F6R
F6/F5R
F7/B1
F8/B2
B3
F1
F2/F8R
F3/F7R
F4/F6R
F5/F5R
F6/B1
F7/B2
F8/B3
legend
forward (left-to-right)
backward (standard)
bidirectional overlap
bubble (warmup only)
bubbles do not grow
with micro-batch count
bubble fraction vs schedule — lower is better
P=8, M=16
1F1B: 30.4% Zero Bubble: 11.3% DualPipe: 5.5% DualPipeV: 6.5%
P=8, M=64
1F1B: 9.9% Zero Bubble: 3.4% DualPipe: 1.5% DualPipeV: 1.8%
P=16, M=128
1F1B: 10.5% Zero Bubble: 3.9% DualPipe: 1.8% DualPipeV: 2.1%
DualPipe cost
2x parameter copies per rank — affordable at MoE scale (EP dominates)
DualPipeV cost
1x parameter copies, slightly larger bubble — better fit for dense models
DeepSeek-V3
2048 H800 GPUs, ~2.8M GPU-hours. DualPipe recovers ~245k GPU-hours
over a 1F1B baseline — enough to fund a separate 70B dense run.