The order to change SGLang settings
Seven settings in the order this manual changes them: measure with sglang.benchmark.serving, size the KV pool, order the prompt, cap concurrency at the knee, then CUDA graphs and attention backends, then parallelism, and disaggregation last. Each row names the section that measures it.
1
measure with sglang.benchmark.serving
TTFT, TPOT, and output tok/s at three concurrency levels · 5.1
2
--mem-fraction-static
size the KV pool for the machine you share · 3.2
3
prompt order
shared text first, per-request text last, read cached_tokens · 3.4
4
--max-running-requests
cap concurrency at the knee you measured · 5.1
5
--cuda-graph-backend-decode, --attention-backend
one change per run, after steps 1 to 4 · 4.1, 4.2
6
--tp-size, --dp-size, --attn-dp-size
only when one replica is saturated · 6.1
7
--disaggregation-mode
only when prefill measurably delays decode · 6.3