One 2,048-token budget per step: decode first, prompt chunks fill the rest
Three scheduler steps with the default budget of 2,048 tokens. Eight running requests take one decode token each, and a new 4,096-token prompt fills the rest in chunks of 2,040 tokens. In step 3 the last 16 prompt tokens run, and the new request decodes from step 4.
BUDGET OF EACH STEP, --MAX-NUM-BATCHED-TOKENS 2048
step 1
2,040 prompt tokens of the new request
step 2
2,040 prompt tokens of the new request
step 3
the last 16 prompt tokens, decode starts in step 4
decode, one token for each running request
a chunk of a new prompt