The failures this manual tells as stories, with their symptom, their cause, their fix or status, and the section that explains each one.
The sections use real incidents as evidence. Each one comes from an SGLang or Miles issue, a pull request, a field report, or our own capture. This index lists them by part, so you can search it with a symptom you see in your own logs. A status of open or closed was checked on 2026-10-16.
Our capture
| Incident | Symptom | Cause | Fix or status | Section |
|---|---|---|---|---|
| Out of memory at concurrency 32 (log) | Metal reported Insufficient Memory, and the whole server process tree exited |
The pool took 87 percent of the free memory that startup saw | --mem-fraction-static 0.4 |
3.2 |
--disable-overlap-schedule on MLX (log) |
RuntimeError on the first real batch, then the server exited through SIGQUIT and killed its process tree |
The path without overlap still calls FutureMap.stash with tensors on two devices |
Open at 0.5.22. Keep overlap on | 2.4 |
| Process-group SIGTERM (log) | The log reports a crash at shutdown | The scheduler child died first, so the parent took the crash path without a drain | Send SIGTERM to the main process only | 7.1 |
| Memory guard on a GB10 (record) | Startup stopped while the scheduler allocated the KV pool. We did not keep the server logs of those starts | A guard daemon of the machine stopped the scheduler, which grew up to 36.7 GiB in 5 s | Stop or configure the guard for the server | 3.2 |
Driver NV_ERR_NO_MEMORY on a GB10 (record) |
Bursts of driver out-of-memory lines in the kernel log | The bursts fall on 2026-10-16, the day of our first starts. We did not keep the server logs of that day, so we cannot match each burst to one start | Every server in our later captures became healthy. Alert on health, not on this line | 3.2 |
| pip install on an L4 host with no CUDA toolkit (record) | Four failed starts: Python.h, nvcc, FlashInfer headers, then -lcudart |
Kernel compilers at startup need a complete toolchain | Use the image or a host with a full CUDA toolkit | 7.1 |
Benchmark with HF_HUB_OFFLINE=1, log not kept |
The random dataset failed with every model on disk |
The dataset downloads ShareGPT from the Hugging Face Hub | --dataset-name random-ids or a local --dataset-path |
7.1 |
--chunked-prefill-size 8192 on an L4 (record) |
Startup ended with a PyTorch INTERNAL ASSERT FAILED |
The prefill graph for 8,192 tokens needed more than the 2.33 GB that the KV pool left | Keep the default chunk of 4,096 on a 24 GB card | 2.2 |
--tp-size 2 with a shared second L4 (record) |
RuntimeError: The memory capacity is unbalanced at startup |
Two pods of another user held 3.5 GB of the second GPU | Use GPUs that nothing else holds, or one GPU | 6.1 |
| Prefill and decode on two L4s over TCP (record, summary lines) | The benchmark counted every request as a success, with 0 retokenized output tokens and a mean TTFT of 0.00 ms | The transfer queue to a small decode pool filled, and the transfer session failed | Read Total generated tokens (retokenized) in every disaggregated benchmark |
6.3 |
| vLLM FlashInfer sampler on the L4 (record) | Engine core initialization failed after a JIT build |
The CUDA compiler and the CUDA toolkit headers in the install did not match | VLLM_USE_FLASHINFER_SAMPLER=0 for a greedy benchmark |
5.3 |
Miles with --attn-implementation triton (log) |
ValueError: Specified attn_implementation="triton" is not supported |
The trainer side does not accept the Triton backend | Use a contract that both sides support | 7.2 |
| Miles true on-policy on a GB10 (log) | SGLang stopped with the FA3 SM assertion, then the job logged about 200,000 503 errors over 63 minutes and never failed |
FA3 accepts SM 8.0 to 9.0, and the rollout loop retries a dead engine | Watch the SGLang log, and check the contract kernels for your GPU | 7.2 |
| GB10 shutdowns under Miles load (record) | The machine shut down three times on 2026-10-18 | The logs do not record a cause | A script pauses the container when the GPU reaches 84 C | 7.2 |
| EAGLE-3 draft for Qwen3-30B-A3B | Startup refused: the target context length of 262144 is greater than the derived 2048 | The draft config declares 2,048 positions | SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1, as the SGLang cookbook sets it. The Qwen3-8B draft declares 40,960 positions and starts without it |
4.4 |
| Thermal throttle in an L4 sweep (record) | CUDA graphs showed no gain, and seeds of one point differed by 49 percent | GPU 0 ran at 87 °C and 360 MHz of 2,040 MHz, with thermal slowdown active | Record GPU clocks and temperature with every run | 5.1 |
| Temperature 0 on a default server (log) | 2 different answers in 5 identical requests on a GB10 | Not isolated in our runs | --enable-deterministic-inference gave 1 answer in 5 |
4.6 |
Part 1 and Part 2: the request path
| Incident | Symptom | Cause | Fix or status | Section |
|---|---|---|---|---|
| LinkedIn field report, 2026-02 | P99 latency of seconds on small ranking models | Tokenization, ZMQ messages, garbage collection, the GIL, and the scheduler | Batch tokenization, a scoring path, and gc.freeze after warm-up (pull request 9241) |
1.2, 2.1 |
| #36889 | --max-running-requests 8 accepted, but throughput flat at 2 streams |
The Mamba state cache capped running requests to 2 | Only an info line reports the cap | 2.2 |
| #8463 | KV cache pool is full. Retract requests. again and again, then failed health checks |
Retraction could not free enough memory and repeated | Closed without a fix. At 0.5.22 the scheduler aborts the last request | 2.3 |
| #26248 | Decode fell from about 429 to 20 to 38 tokens per second after a retraction | Not recorded | Closed for inactivity | 2.3 |
| #14972 | Allocation failure after KV cache pool is full at 94 percent usage, with overlap on |
Retraction did not account for requests that finish one step later | Pull request #15626 closed without merge. The 0.5.22 scheduler handles it | 2.4 |
| #34740 | Detokenization work grew with the square of the output length | Byte tokens that never complete a character never advance the offsets | Open | 2.5 |
Part 3: the KV cache
| Incident | Symptom | Cause | Fix or status | Section |
|---|---|---|---|---|
| #39675 | Out of memory on the first prefill on Apple MPS | Unknown capacity made the fraction 0.95 | Set --mem-fraction-static |
3.2 |
| #42510 | The draft pool allocation failed | Draft-model copies stayed in memory while the target pool was sized | Reported | 3.2 |
| #6027 | A 64K-token DeepSeek-V3 prompt ran out of memory in attention | The log reported room for 274,934 KV tokens, and attention still ran out | Lower the fraction | 3.2 |
| #43201 | Older posts describe a ChunkCache class that no longer exists |
The class was deleted on 2026-10-08 | --disable-radix-cache builds a disabled UnifiedRadixCache |
3.3 |
| #32459 | Deep-prompt hit rate fell from 97 to 40 to 53 percent with EAGLE | The extra KV for speculation forced the tree to evict prompts | Measure the hit rate with each feature you add | 3.3 |
| #35129 | Hit rate of 66.8 percent, and HiCache load-back evicted other sessions | A random hash and a rotating message in the system prompt, and load-back with no room | Prompt edits moved the hit rate to 94.4 percent | 3.4, 3.5 |
| Jarvislabs field report, 2026-08 | 23 percent of input tokens computed again | Requests went to servers without their prefix | One header pinned each conversation to one GPU: 64.7 to 92.1 percent hits | 3.4, 6.4 |
| #30760 | One tensor-parallel rank waited forever in a prefetch check while the others moved on | Not recorded | Reported | 3.5 |
| #42465 | Bursts of 540,000-token prompts hung every rank with HiCache on | Not recorded | The same prompts run without HiCache | 3.5 |
| #39830 | Fluent answers to a different prompt on 20 of 20 host-tier hits | A HiCache host-tier hit on a hybrid linear-attention model | Test hybrid models before you trust the cache | 3.6 |
| #22935 | An identical prompt reused no tokens | A node split created a node with no Mamba state | Read #cached-token for repeated prompts |
3.6 |
| #37834 | Hit rate fell from 88.2 to 65.9 percent, and TTFT rose from 226 ms to 1,063 ms | An option forced the no_buffer Mamba strategy |
Keep --mamba-radix-cache-strategy auto unless you measured better |
3.6 |
| #36131 | Lower single-stream decode for hybrid SWA models | A new default freed SWA slots outside the window | Reported | 3.6 |
Part 4: each forward pass
| Incident | Symptom | Cause | Fix or status | Section |
|---|---|---|---|---|
| #23769 | DeepSeek-V4-Pro-FP8 exited during graph capture | With attention data parallelism, the graph batch size counts per rank | A smaller graph batch size or a lower fraction | 4.1 |
| #36478 | disable_piecewise_cuda_graph=True raised TypeError in the Python API |
Pull request #23906 moved the field into the per-phase graph config | cuda_graph_backend_prefill="disabled" |
4.1 |
| joshua8.ai field report, 2026-01 | SGLang ran on an SM120 card only in eager mode, and NVFP4 did not run | New architecture support | Check capture and the weight format before you compare engines | 4.1, 4.3 |
| #17411 | Median ITL on H20 went from 39.44 ms to 283.52 ms | FlashInfer 0.6.1 regressed the Hopper attention kernel | SGLang made fa3 the Hopper default |
4.2 |
| Cloudflare field report, 2026-08 | A BF16 KV cache ran out of memory at concurrency 64 | The cache did not fit | An FP8 KV cache held twice the tokens | 3.1, 4.3 |
| Pivot News summary, 2026-05 | A mixed cluster crashed when RTX 4090 cards joined | The FP4 path required compute capability 10.0 | No software path for older cards | 4.3, 5.3 |
| #5274 | EAGLE-3 lowered throughput at concurrency 20 from 217.8 to 164.4 tokens per second | Speculation spends compute that a full batch needs | Run speculation at small batch | 4.4 |
| #6783 | Acceptance fell to 1.2 tokens at 10K input | The draft config had 2,048 positions and no rope scaling | Add rope scaling and 131,072 positions to the draft config | 4.4 |
| David Noel Ng field report, 2026-08 | Acceptance of 1.3 to 1.8 tokens on DeepSeek-V4-Flash | The draft loader did not map the shared expert weights | Pull request #33276 | 4.4 |
| #37707 | A string field accepted values shorter than its minLength |
xgrammar dropped the rule when the field also had a pattern |
Open | 4.5 |
| #38863 | Bitmask CPU time grew with output length under speculation | A rollback method copied a debug list on every call | Open | 4.5 |
| #43055 | gpt-oss-20b gave different greedy outputs in prefix mode | The deterministic Triton kernel aligned KV tiles to the trimmed prefix | Open | 4.6 |
| #43413, #36174, #31261 | Deterministic mode does not cover Mamba2 hybrids, DeepSeek-V4, or Qwen3.5 | Architectures without batch-invariant kernels | Not yet supported | 4.6 |
| #39597 | Greedy text changed after a restart with DFLASH | FlashInfer autotuning picked other kernel tactics | --disable-flashinfer-autotune or deterministic mode |
4.6 |
Part 5: measuring
| Incident | Symptom | Cause | Fix or status | Section |
|---|---|---|---|---|
| #3196 | 13 tokens per second for DeepSeek V3 on eight H200 GPUs | The script counted words and included TTFT in the rate | Measure with the benchmark client | 5.1 |
sglang:utilization |
The gauge reads 0 on a busy server | A regression that the code comment links to #22713 | Use the queue and latency series | 5.2 |
| #35884 | Failed health probes added load | A timed-out /health call did not cancel its request |
Treat a failing probe as one symptom | 5.2, 7.1 |
| #22511 | "health check failed" under high load on GLM-5 | No output reached the detokenizer for 20 s under load | Use /ready for readiness |
5.2, 7.1 |
| AIMultiple, 2026-01 | "SGLang 29% faster" repeated by at least five posts | One offline test that lists SGLang v0.2.3 | Ignore results without a version | 1.3, 5.3 |
| vLLM discussion #17221 | SGLang appeared to use 7 GB against 21 GB | The wrong flag: --context-length matches --max-model-len |
Compare engines with matching flags | 5.3 |
Part 6 and Part 7: fleets and operations
| Incident | Symptom | Cause | Fix or status | Section |
|---|---|---|---|---|
| #19953 | Attention data parallelism across nodes hung on its first message | A ZeroMQ socket bound without IPv6 for an IPv6 --dist-init-addr |
A fix was proposed. We could not confirm the merge | 6.1 |
| #9787 | One prefill GPU went idle, then every prefill process crashed | Imbalanced routing sent too many tokens to one GPU in DeepEP dispatch | Smaller chunks, a tuned expert placement, and EPLB | 6.2 |
| #12591 | TTFT with disaggregation was higher than on one server | An old Mooncake transfer engine | Upgrade Mooncake | 6.3 |
| #9266 | "Prefill bootstrap failed" at 64 concurrent requests | The router timeout for the prefill client was too short | Pull request #9803 | 6.3 |
| #9926 | "Aborted by AbortReq" errors | The client or the router aborted | Check timeouts before the transfer | 6.3 |
| Novita AI field report, 2026-01 | TTFT got much worse with disaggregation and overlap | The KV transfer waited for the next kernel launch | Their flag is not in the 0.5.22 code | 6.3 |
| #22095 | A two times throughput regression on GB200 NVL72 | The router outside SGLang changed version | Pin and benchmark the router with the engine | 6.4 |
| #10608 | "Failed to select prefill/decode worker" with cache_aware |
Routing with disaggregation and Kubernetes discovery | Users switched to round robin | 6.4 |
| CERIT-SC field report, 2026-04 | Tool calls in reasoning output, rare hangs, and loops that held KV | Agentic traffic with very long prompts | A 32K output cap and a presence penalty | 7.1 |
| Doubleword field report, 2026-04 | A cold start of 695 s | 531 s of weight loading and cold kernel caches | Persistent caches and a process checkpoint: 9.6 s | 7.1 |
| radixark/miles#860 | /release_memory_occupation did nothing for sliding-window models |
The SWA pool turns the memory saver off | Open | 7.2 |
| radixark/miles#1485 | down_proj outputs differed at tensor parallel 2 |
The two engines summed BF16 partial results in different orders | Open | 7.2 |
| radixark/miles#3714 | Weight transfer failed with Message truncated |
Deterministic serving pinned the NCCL channel limit to 8, and the trainer used 24 | Open | 7.2 |
| radixark/miles#3277 | New MoE routes next to old log probabilities after a retract | Re-prefill under new weights overwrote the recorded routes | Open | 7.2 |
Sources: the incident paragraphs of every section under sections/; capture/out/oom-exploratory.log, server-no-overlap.log, server-default.log; capture/gpu/incidents/gb10-memory-guard-pauses.txt, gb10-driver-no-memory.txt, l4-toolchain-failures.txt, l4-thermal-throttle.txt; capture/gpu/out-gb10/determinism.txt and deterministic-determinism.txt; docs/cookbook/specbundle/specbundle_usage.mdx at v0.5.22; the GitHub issues and pull requests linked in the tables, capture/gpu/incidents/l4-chunk-8192-prefill-graph-oom.txt, capture/gpu/incidents/l4-tp-unbalanced-memory.txt, capture/gpu/incidents/l4-pd-mooncake-tcp-queue.txt, capture/gpu/incidents/l4-pd-summary-lines.txt, capture/gpu/incidents/l4-vllm-flashinfer-jit.txt, capture/gpu/incidents/gb10-boot-list.txt, capture/gpu/out-miles/true-on-policy-error.log, capture/gpu/out-miles/true-on-policy-fa3-error.log