One /generate request through the three processes
Four lifelines: the client, the tokenizer manager, the scheduler, and the detokenizer. The client posts a five-token prompt. The tokenizer manager sends token ids to the scheduler over ZMQ. The scheduler matches two cached tokens in the radix cache, prefills three, and returns the first token after 68 ms. Eleven decode steps follow, and the detokenizer returns the text, for 725 ms end to end.
client
TokenizerManager
Scheduler
Detokenizer
1
POST /generate
The capital of France is
2
5 token ids
one ZMQ message
3
match_prefix
2 tokens cached
4
prefill
3 new tokens
5
first token
68 ms after the request
6
11 more tokens
decode steps, 657 ms
7
text
Paris. The capital of Italy is
8
200 OK
e2e_latency 0.725 s