This page lists the flags the manual uses, grouped by user job, with the default at 0.31.0.
Use this page to look up a flag that a section names.
Defaults are for vllm serve at 0.31.0 on a CUDA GPU, read from the configuration classes in the source.
Server
| Flag |
Default |
What it sets |
Page |
--host |
not set, so every network interface |
the address the server listens on |
server options |
--port |
8000 |
the HTTP port |
server options |
--served-model-name |
the model name |
the name that clients send in model |
server options |
--api-key |
not set |
the key that the /v1 routes require |
server options |
--chat-template |
the template of the model |
the chat template, as a file or as text |
server options |
--default-chat-template-kwargs |
not set |
template values for every request |
reasoning models |
--reasoning-parser |
not set |
the parser that moves thinking out of the answer |
reasoning models |
--enable-log-requests |
off |
one log line for each request |
server options |
Memory and length
| Flag |
Default |
What it sets |
Page |
--gpu-memory-utilization |
0.92 |
the share of total GPU memory for one instance |
startup |
--kv-cache-memory-bytes |
not set |
the KV cache size in bytes, which overrides the share for the cache |
startup |
--max-model-len |
the model config |
the longest prompt plus answer for one request |
long contexts |
--block-size |
16 |
tokens in one KV block, or the attention backend's preference |
block pool |
--kv-cache-dtype |
auto |
the data type of the KV cache, such as fp8 |
attention backend |
--num-gpu-blocks-override |
not set |
a fixed number of KV blocks, for tests |
startup |
Scheduling
| Flag |
Default |
What it sets |
Page |
--max-num-batched-tokens |
2048 for vllm serve on a GPU under 70 GiB or an A100, 8192 from 70 GiB, 16384 from 160 GiB |
the token budget for one step |
token budget |
--max-num-seqs |
256 on a GPU under 70 GiB or an A100, 1024 from 70 GiB |
the most requests that run at once |
token budget |
--enable-chunked-prefill |
on |
splitting a long prompt across steps |
token budget |
--async-scheduling |
on for most generate models |
planning the next step while the GPU runs |
Model Runner V2 |
--scheduling-policy |
fcfs |
first come first served, or priority |
token budget |
--long-prefill-token-threshold |
not set |
a cap on the tokens of one prefill in one step |
token budget |
Cache
| Flag |
Default |
What it sets |
Page |
--enable-prefix-caching |
on |
reuse of cached prefix blocks |
block pool |
--prefix-caching-hash-algo |
sha256 |
the hash of each block |
block pool |
--disable-hybrid-kv-cache-manager |
off |
one group for every layer kind |
long contexts |
--kv-transfer-config |
not set |
a KV connector and its role |
KV connectors |
Execution
| Flag |
Default |
What it sets |
Page |
-O0 to -O3 |
-O2 |
the compile and CUDA graph level |
compile and graphs |
--enforce-eager |
off |
no compile, no CUDA graphs, and no kernel warmup |
compile and graphs |
--max-cudagraph-capture-size |
set by the level |
the largest batch with a captured graph |
compile and graphs |
--attention-backend |
chosen from a list |
the attention kernel, by enum name |
attention backend |
--speculative-config |
not set |
the speculation method and its options, as JSON |
speculative decoding |
Parallelism
| Flag |
Default |
What it sets |
--tensor-parallel-size, -tp |
1 |
GPUs that split each layer |
--pipeline-parallel-size, -pp |
1 |
GPUs that split the layers into stages |
--data-parallel-size, -dp |
1 |
full copies of the model |
--enable-expert-parallel |
off |
experts spread across GPUs for mixture-of-experts models |
Environment variables
| Variable |
Default |
What it sets |
VLLM_ENABLE_V1_MULTIPROCESSING |
1 |
the engine core in its own process |
VLLM_USE_V2_MODEL_RUNNER |
not set |
a pinned choice of model runner |
VLLM_BATCH_INVARIANT |
0 |
the batch-invariant mode, when set to 1 |
VLLM_API_KEY |
not set |
the API key, as --api-key sets it |
VLLM_LOGGING_LEVEL |
INFO |
the vLLM log level |
VLLM_NO_USAGE_STATS |
0 |
no usage statistics, when set to 1 |
Sources: the vLLM source at v0.31.0 (vllm/config/cache.py, vllm/config/scheduler.py, vllm/config/compilation.py, vllm/config/model.py, vllm/config/parallel.py, vllm/engine/arg_utils.py, vllm/envs.py, vllm/entrypoints/launchers/cli_args.py, vllm/config/structured_outputs.py).
Code and artifacts 1
What you take away
Saved in your notebook on this device.
Sources and limits
Reading checkpoints are self-reported. Test results are recorded separately and apply to the executed cases.
AI Engineering from Scratch · 3f9715c · MIT