Keep CUDA graphs and compile on

The default compile level is the right one for most servers, and eager mode costs a few percent of decode speed.

Startup takes minutes and you want it faster. When you finish this section, you can decide what each compile level and eager mode cost.

Our result: on our L4, eager mode cost about 2 percent of the time per token in a cooled run. In a throttled run, -O0 cost about 2 percent for one user, and -O3 matched the default.

Know the switches

vLLM sets compile and CUDA graphs with one level, from -O0 to -O3. The default is -O2. The compile page explains what each part does.

Read what the start costs

The startup log gives the time of each part. On our DGX Spark, with Qwen3-30B-A3B, the two graph captures and the compile took about 96 seconds of a start of about eight minutes:

(EngineCore pid=143) INFO 10-21 10:59:45 [model_runner.py:1087] Graph capturing finished in 39 secs, took 1.17 GiB
(EngineCore pid=143) INFO 10-21 11:00:51 [model_runner.py:1087] Graph capturing finished in 38 secs, took 0.31 GiB
(EngineCore pid=143) INFO 10-21 11:00:53 [core.py:373] init engine (profile, create kv cache, warmup model) took 127.54 s (compilation: 18.67 s)

title: graph capture and compile time at startup
source: capture/gpu/out-gb10/server-load-vllm.log
lang: text
note: The first two lines are the two capture passes. The last line gives the whole engine start and the compile share.
---
(EngineCore pid=143) INFO 10-21 10:59:45 [model_runner.py:1087] Graph capturing finished in 39 secs, took 1.17 GiB
(EngineCore pid=143) INFO 10-21 11:00:51 [model_runner.py:1087] Graph capturing finished in 38 secs, took 0.31 GiB
(EngineCore pid=143) INFO 10-21 11:00:53 [core.py:373] init engine (profile, create kv cache, warmup model) took 127.54 s (compilation: 18.67 s)

On that start, loading the weights took about 330 seconds, much more than the compile. A later start can read the compile cache, so the compile share gets smaller.

See what our runs show

We measured eager mode against the default on the L4 with Qwen/Qwen3-8B on 2026-10-21, in the full-clock session. Both runs used the same pool, seven concurrencies, and three seeds, with the GPU cooled before each run.

Fig. 4.3eager mode against the defaultchart

Original SVG

Original source

id: fig-4-3
kind: chart
title: eager mode against the default
claim: On our L4, eager mode was 1.2 to 2.3 percent slower per token at every concurrency.
caption: Bands span the lowest and highest of three runs. The two lines sit close because an 8B decode step waits on memory, not on kernel launches. From capture/gpu/summary/l4.json.

The seed ranges did not overlap from 1 to 32 users, so the gap is real, and it is small. On our DGX Spark, eager mode cost 2 to 6 percent of the time per token.

We then ran the compile levels on 2026-10-22, in the throttled session, at 1 and 32 users. The default had three seeds and the other two levels had two. The GPU was throttled for all three: the median clock was 210 MHz at one user and 450 to 548 MHz at 32 users.

At one user, -O0 was 2.2 percent slower, and the seed ranges did not overlap. At 32 users, the three levels overlapped. -O3 matched the default, as the code says it should at this version. Because of the low clock, read these as a comparison inside one session, not as speeds you will see.

Decide

Keep the default for production. The graphs and the compile cost about one and a half minutes on our DGX Spark start, and the cache makes later starts faster.

When you test a change and restart often, use -O0. It skips the compile and the graph capture, and it cost us about 2 percent of decode speed. Use --enforce-eager only to debug a kernel, because it also skips the warmup, so the first requests after a start are slow.

- Leave the default compile level on for production.
- Use -O0, not eager mode, when you only want a faster start.
- Expect -O3 to equal the default in vLLM 0.31.0.

Sources: capture/gpu/summary/l4.json (load.vllm-b, load.vllm-eager, load.vllm-fast-b, load.vllm-O0, load.vllm-O3), capture/gpu/summary/gb10.json (load.vllm, load.vllm-eager), capture/gpu/out-gb10/server-load-vllm.log, vLLM documentation at v0.31.0 (docs/design/optimization_levels.md), the vLLM source at v0.31.0 (vllm/config/model.py).