Split rollout and training across GPUs · Ingénierie IA

On many GPUs, rollout and training can run on separate devices, and a transfer method moves the weights between them.

On one GPU, the trainer and the engine take turns, and the weight update is a copy inside the GPU. On many GPUs you have a choice. Keep both sides on the same GPUs and take turns, or give each side its own GPUs and send the weights across.

When you finish this section, you can choose a placement and a weight transfer method for a run on many GPUs.

We cannot measure this on our single GB10. This page comes from the Miles source and documentation.

Choose a placement

Placement	Flag	GPUs used	Each side waits for the other
Colocated	`--colocate`	the larger of the trainer GPUs and the rollout GPUs	yes, every step
Split	no `--colocate`	the trainer GPUs plus the rollout GPUs	yes, unless you run async training

Set the sizes with these flags:

--actor-num-nodes and --actor-num-gpus-per-node set the trainer GPUs.
--rollout-num-gpus sets the rollout GPUs in a split layout.
--rollout-num-gpus-per-engine sets how many GPUs each rollout engine uses.

Start colocated. Split only when the rollout phase takes most of the step and you have the GPUs to give it. A split layout leaves trainer GPUs idle during rollout unless you also run async training, as in section 4.1.

Choose how the weights move

In a split layout, the weights leave the trainer GPUs after every update. --update-weight-transfer-mode picks the method:

Method	What it does	Needs
`broadcast`, the default	sends each tensor over NCCL, one by one	nothing extra
`broadcast_packed`	packs each bucket into one send	the Megatron trainer and a split layout
`p2p`	moves the weights over RDMA with Mooncake	Mooncake and an RDMA network
`disk-delta`	writes only the changed bytes to a shared folder, and each engine loads them	`--update-weight-disk-dir`

Under --colocate, Miles uses a copy inside the GPU instead, and the transfer mode does not apply.

Choose the trainer

This manual uses the FSDP trainer, because it loads a Hugging Face model folder as it is. Miles has two other trainers, and --train-backend picks one:

Trainer	Model input	Model parallel	LoRA
`megatron`, the default	a Megatron `torch_dist` checkpoint, converted once from the Hugging Face folder	tensor, pipeline, context, and expert parallel	yes
`fsdp`	the Hugging Face folder	no. It shards the weights and the optimizer state over data-parallel GPUs, and each GPU still runs whole layers	no
`torchtitan`	the Hugging Face folder, for the models that torchtitan implements	tensor, pipeline, context, and expert parallel	no

Some features in this manual work only on Megatron: fully async training with --colocate, train fault tolerance, MTP training, and the step-0 checks of --ci-test. The Miles documentation recommends Megatron for large models and for jobs on many nodes.

torch_dist: the checkpoint format of Megatron. It does not depend on the parallel layout, so you can change the layout later without a new conversion.

Convert a model once before the first Megatron run. The architecture flags come from the model file of Miles for that model, here qwen3-4B:

bash

cd /root/miles
MODEL_ARGS=$(python3 miles/utils/external_utils/model_args_utils.py qwen3-4B)
PYTHONPATH=/root/Megatron-LM python3 tools/convert_hf_to_torch_dist.py $MODEL_ARGS --hf-checkpoint /root/lab/models/Qwen3-4B --save /root/lab/models/Qwen3-4B_torch_dist

Run these lines inside the container, with the lab folder mounted as in section 1.1. Our launcher builds an FSDP run only. For Megatron, start from a launcher of Miles for your model, in the scripts folder of the Miles source. We have not run the Megatron or the torchtitan trainer on our GB10. We ran only the conversion to torch_dist for Qwen3-0.6B, and it wrote the checkpoint folder.

Start a run on more than one node

Every node must be in one Ray cluster before Miles submits the job. The launchers of Miles do it in one of two ways:

One command for each node role. A worker command joins the cluster of the head node and waits. A train command starts the head node, waits for all GPUs, and submits the job.
One command on the head node, which connects to the other hosts over ssh and adds them to the cluster.

To use a Ray cluster that you started yourself, set MILES_SCRIPT_EXTERNAL_RAY=1. The launcher then skips its own ray start and only submits the job. Set MASTER_ADDR to the address of the head node on every node, so that Ray and the trainer use the right network interface. Then set the trainer size with --actor-num-nodes and --actor-num-gpus-per-node.

Section 6.3 lists a known problem of some launchers with a Ray cluster of your own. We have not run Miles on more than one node.

What to measure after the split

perf/update_weights_time against perf/step_time. On many machines the transfer is no longer a copy inside one GPU.
The idle time on each side, from perf/train_wait_time and the rollout timers.
The gap between the trainer and the engine, as in section 5.2, after the first update with the new method.

Original source

- Start colocated, and split only when rollout takes most of the step.
- Keep broadcast until the weight transfer is a large share of the step.
- Measure the update time and the idle time on both sides after a split.
- Pick the trainer before you size the run, because only Megatron has model parallel and LoRA.

Sources: Miles main 08752a12 files docs/user-guide/training-backend.md, docs/user-guide/launch-script.md, miles/utils/external_utils/command_utils/ray_backend/backend.py, miles/utils/lora/arguments.py, tools/convert_hf_to_torch_dist.py, scripts/models/qwen3-4B.py, miles/ray/placement_group.py, miles/utils/arguments.py, miles/backends/training_utils/weight_update/protocol.py, miles/backends/training_utils/weight_update/protocols/broadcast.py, miles/backends/training_utils/weight_update/protocols/delta.py, miles/backends/training_utils/metrics/perf.py, docs/advanced/disaggregated-rollout.md, docs/advanced/p2p-weight-transfer.md; capture/gpu/out-gb10/live-check-summary.txt (live check, 2026-10-23).