A larger model costs more to offload than to update
Mean seconds of steps 2 and 3 of 4 short steps, after two warm-up steps. Weight update: 0.64 s for Qwen3-0.6B and 0.89 s for Qwen3-1.7B. Trainer offload: 1.14 and 2.92 s. Trainer onload: 0.81 and 1.44 s.
0
2
4
seconds, mean of steps 2 and 3
0.64 s
update 0.6B
0.89 s
update 1.7B
1.14 s
offload 0.6B
2.92 s
offload 1.7B
0.81 s
onload 0.6B
1.44 s
onload 1.7B
Qwen3-0.6B
Qwen3-1.7B, 2.8 times the size
The update grew 1.4 times. Trainer offload and onload grew 2.6 and 1.8 times.