One weight update between Miles and SGLang
Miles rank 0 pauses SGLang in retract mode, opens a weight-update session, sends each bucket as an HTTP description followed by an NCCL broadcast, closes the session so quantized layers are repacked once, sets the new weight version, and resumes generation.
Miles rank 0
SGLang server
1
POST /pause_generation
mode retract: running requests return to the queue
2
POST /begin_weight_update
layers prepared for checkpoint-format writes
3
POST /update_weights_from_distributed
one bucket: names, dtypes, and shapes
4
NCCL broadcast from rank 0
repeat rows 3 and 4 for every bucket
5
POST /end_weight_update
quantized layers repacked once
6
POST /update_weight_version
later tokens carry the new version
7
POST /continue_generation
retracted requests prefill again