Where the time of one step goes
Median seconds per phase over 48 steps of a run made before we added the heat guard, Qwen3-0.6B on one GB10: rollout 47.5, log probabilities 23.4, gradient step 77.0, which together with the log probabilities is the training phase of 100.3, trainer offload 0.76, trainer onload 0.87, and weight update 0.64.
0
30
60
90
seconds per step, median
47.5 s
rollout
23.4 s
log-probs
77.0 s
gradient step
0.76 s
offload
0.87 s
onload
0.64 s
update
Tune training and rollout first. The three amber phases add up to about 2 s.