The engine and the trainer take turns on one memory
One colocated step drawn to scale from the phase medians of 48 steps of a run made before we added the heat guard. The engine holds its weights and its KV cache during rollout, 47.5 s, then gives them back. The trainer moves its state onto the GPU in 0.87 s, trains in 100.3 s, of which 23.4 s compute the log probabilities, and moves its state off in 0.76 s. The engine loads again and gets the new weights in 0.64 s. Over the same 48 steps, a median of 2.2 s of each 152.6 s step has no timer.
PHASE MEDIANS OF 48 STEPS, TO SCALE
engine
weights, KV cache
rollout 47.5 s
trainer
weights, optimizer
train 100.3 s, with 23.4 s of log-probs
trainer onload 0.87 s
trainer offload 0.76 s
then the update 0.64 s
Only one side holds the GPU memory at a time.
The timed hand-offs take about 2 s of a step of about 2.5 minutes.
A median of 2.2 s of each step has no timer, such as the engine release.