One constrained request from arrival to sampling A request with a json_schema arrives at the grammar manager, which looks up the key json plus schema in the grammar cache. On a hit the request goes straight to the waiting queue. On a miss, compilation runs in a thread pool and the request waits in the grammar queue until the grammar is ready on every rank. During each decode step the grammar fills a vocabulary bitmask that masks the logits before the sampler picks a token, and the grammar then accepts the token. request sampling_params.json_schema grammar cache key ("json", schema) waiting queue can be scheduled hit thread pool compiles, gives a future grammar queue ready on every rank? miss ready logits one row per request vocab bitmask invalid tokens removed sampler picks a valid token EVERY DECODE STEP after sampling, the grammar accepts the token and moves its state