One streamed request with a stop string
Four lifelines: client, TokenizerManager, scheduler, and detokenizer. The client posts a streaming request, the scheduler sends token ids to the detokenizer, which sends text back for the client as SSE chunks. The scheduler holds back text whose tail could start a stop string, then matches the stop string and finishes, and the detokenizer trims the stop text before the last chunk.
client
TokenizerManager
scheduler
detokenizer
1
POST, stream true
2
tokenized request
3
new token ids
4
new text
5
SSE chunk
6
holds a possible stop prefix
7
stop string matched, finish
8
last ids, finish_reason stop
9
text with the stop trimmed
10
last chunk, then [DONE]