Split prefill and decode: the KV cache moves between two servers
A router sends a new request to a prefill server, which computes the prompt and writes its KV cache. The NixlConnector copies the cache to a decode server, which then streams the tokens back. Both servers set a kv_connector in --kv-transfer-config.
router
picks a pair
prefill server
computes the prompt
decode server
writes the tokens
NixlConnector
copies KV blocks
Both servers: --kv-transfer-config with "kv_connector": "NixlConnector".