One request through the three vLLM processes
A client sends HTTP to the API server process, which tokenizes and sends the request over ZeroMQ to the engine core process. The engine core schedules it and sends the batch to a GPU worker, one for each GPU. Token ids come back the same way, and the API server streams text to the client.
YOU
PROCESS 1
PROCESS 2
ONE FOR EACH GPU
client
curl, OpenAI
SDK, bench
API server
HTTP routes
tokenizer
engine core
Scheduler
KVCacheManager
GPU worker
GPUModelRunner
forward pass
1
2
3
1
HTTP request, text in
2
ZeroMQ message, token ids
3
shared memory queue, one batch per step
token ids back, one step at a time
text streamed to the client
A worker crash shows up in the engine core log. A slow tokenizer shows up as slow first tokens.