One setting, the GPU memory fraction, decides most startup failures, and a shorter maximum length or a lower concurrency cap fixes most of the rest.
The server stopped before it printed the KV cache size line. When you finish this section, you can read the reason from the last lines of the log and pick the one flag that fixes it.
Know what vLLM claims at startup
vLLM does four things before it serves a request:
- It loads the weights.
- It runs one profile pass with a full batch to measure the activation memory.
- It captures CUDA graphs.
- It gives the KV cache the rest of its share of the GPU.
The share is --gpu-memory-utilization. The default is 0.92, so vLLM claims 92 percent of the total GPU memory for itself. The number is a limit for one vLLM instance. It is not a fraction of the memory that is free when the server starts. On unified memory, such as a DGX Spark, it is not always a hard limit, as the DGX Spark page explains.
Find your failure in the log
Read the last 30 lines of the server log. Most startup failures match one of three cases.
| What the log says | Cause | Fix |
|---|---|---|
| The KV cache needed for one request at the maximum length is larger than the available KV cache memory | The weights and the profile leave too little for one full-length request | Lower --max-model-len. The error gives the largest length that fits. |
| The free memory on the device is less than the desired GPU memory utilization | Another process holds GPU memory | Lower --gpu-memory-utilization below the free share, or stop the other process |
| CUDA out of memory during the profile run or the graph capture | The batch in the profile pass or the graphs need more memory than is left | Lower --max-num-seqs or --max-num-batched-tokens, or lower --max-cudagraph-capture-size |
Original source
id: fig-2-2
kind: decision
title: where a start fails
claim: The last line before the start fails tells you which of three settings to change.
caption: Read down each column: where the log stops, the cause, and the flag to change. From the vLLM source at v0.31.0 and our runs on the L4.Fix a shared GPU
On a shared GPU, set the fraction yourself. We ran vLLM 0.31.0 on an L4 that another team also used. Their processes held part of the memory, so we set --gpu-memory-utilization 0.83 and a pool of 2,928 tokens.
The first start still failed, because we also set the maximum length to 2,928 tokens. One request at that length needed slightly more KV cache than was free, and the error said that 2,912 tokens would fit:
(APIServer pid=3681321) INFO 10-22 00:35:04 [api_utils.py:286] non-default args: {'model_tag': 'Qwen/Qwen3-8B', 'host': '127.0.0.1', 'port': 30100, 'model': 'Qwen/Qwen3-8B', 'max_model_len': 2928, 'gpu_memory_utilization': 0.83, 'num_gpu_blocks_override': 183}
(EngineCore pid=3682071) ERROR 10-22 00:36:18 [core.py:1433] ValueError: To serve at least one request with the model's max seq len (2928), (0.4 GiB KV cache is needed, which is larger than the available KV cache memory (0.4 GiB). Based on the available memory, the estimated maximum model length is 2912. Try increasing `gpu_memory_utilization` (which also controls CPU memory on the CPU backend) or decreasing `max_modOriginal source
title: the maximum length error on the shared L4 GPU
source: capture/gpu/incidents/l4-gpu1-vllm-maxlen-oom.txt
lang: text
note: The first line gives the settings. The second line gives the error and the length that fits. Our capture cut the second line at 420 characters. In vLLM 0.31.0, the rest of the message tells you to raise the memory fraction or to lower the maximum length.
---
(APIServer pid=3681321) INFO 10-22 00:35:04 [api_utils.py:286] non-default args: {'model_tag': 'Qwen/Qwen3-8B', 'host': '127.0.0.1', 'port': 30100, 'model': 'Qwen/Qwen3-8B', 'max_model_len': 2928, 'gpu_memory_utilization': 0.83, 'num_gpu_blocks_override': 183}
(EngineCore pid=3682071) ERROR 10-22 00:36:18 [core.py:1433] ValueError: To serve at least one request with the model's max seq len (2928), (0.4 GiB KV cache is needed, which is larger than the available KV cache memory (0.4 GiB). Based on the available memory, the estimated maximum model length is 2912. Try increasing `gpu_memory_utilization` (which also controls CPU memory on the CPU backend) or decreasing `max_modWe set --max-model-len 2048, which is enough for our 640-token requests, and the server started. Later we raised it to 2,304 tokens, so that our tests with 2,048-token prefixes fit.
So on a shared GPU, the fraction and the maximum length work together. Lower the maximum length to the longest request you send before you lower the fraction.
Set the KV cache size directly
If you know the KV cache size you want, set it in bytes with --kv-cache-memory-bytes. vLLM then ignores --gpu-memory-utilization for the cache. Use this when two vLLM instances share one GPU and each one needs a known pool.
--num-gpu-blocks-override sets the number of 16-token blocks for a test, as in our fixed pool. Do not use it in production, because it does not check that the memory exists.
Check the fix
Start the server again with tee server.log, as in the first start, and wait for the pool line:
grep "GPU KV cache size" server.logIf the line is there, the server has a pool. Compare its size with your count.
Original source
- Read the last 30 lines of the log, and match the failure to one of three cases.
- On a shared GPU, lower the maximum model length before you lower the memory fraction.
- Use the exact KV memory flag when two instances share one GPU.Sources: capture/gpu/incidents/l4-gpu1-vllm-maxlen-oom.txt, vLLM documentation at v0.31.0 (docs/configuration/optimization.md), the vLLM source at v0.31.0 (vllm/config/cache.py, vllm/config/scheduler.py), our L4 run on the shared GPU on 2026-10-22.