One disaggregated request
Four lifelines: client, gateway, prefill server with its bootstrap server on port 8998, and decode server. The client posts a request to the gateway. The gateway sends it with a bootstrap room to prefill and to decode at the same time. Decode looks up the prefill rank and sends the addresses of reserved KV slots. Prefill runs its forward pass and writes the KV pages into those slots by RDMA. Decode streams tokens to the client.
client
gateway
prefill
bootstrap :8998
decode
1
POST /generate
2
request + bootstrap_room
one random 63-bit number
3
same request, same room
sent at the same time
4
/route, then slot addresses
decode reserves KV slots first
5
KV pages by RDMA write
after the prefill forward pass
6
token stream
decode serves the client