One decode step, launched and replayed Two columns. On the left, eager decode: for each kernel of one transformer layer, the Python model runner sends one launch, and the step repeats this for every layer. On the right, graph replay: the decode CUDA graph runner sends one replay call for a captured batch size, and the GPU runs the recorded kernels of all layers in order. A · EAGER DECODE B · GRAPH REPLAY launch rms_norm launch qkv_proj GEMM launch attention decode launch o_proj GEMM launch gate_up_proj GEMM launch silu_and_mul launch down_proj GEMM one CPU launch for each kernel, repeated for every layer of the model replay(bs=16) one CPU call for the whole step captured decode graph, bs=16 every layer, every kernel same order as column A fixed shapes and addresses recorded once at startup one call a batch of 13 pads to 16 and replays a batch above max_bs cannot replay and runs as column A