--attention-backend

SGLang has many attention kernels because none wins on every GPU and model, and its default choice records regressions that someone already paid for.

In January 2026 a user upgraded FlashInfer and served Qwen3-235B on H20 GPUs. Median inter-token latency went from 39.44 ms to 283.52 ms, seven times slower, and nothing else in the setup had changed. The FA3 kernel gave 56.73 ms on the same run. That report is issue #17411, and the code that picks your attention backend at v0.5.22 still cites it.

When you finish this section, you can name the backend SGLang picks for your GPU and model, and explain why. You can also split prefill and decode across two backends and decide when to override the default.

What an attention backend is

Attention backend: the class that computes attention over the paged KV cache for one batch. Every backend implements the same interface. It builds per-batch metadata such as page tables and index buffers. It prepares state for CUDA graph capture and replay. It runs forward_extend for prefill and forward_decode for decode.

The model code never calls a kernel by name. Each attention layer calls whatever backend the model runner installed at startup. So one flag swaps the kernel for every layer of every model.

At v0.5.22 the flag accepts 25 names. Two are aliases (nsa for dsa, and compressed for dsv4), so there are 23 backends.

The obvious fix, and why it fails

The obvious fix is to read a benchmark, find the fastest kernel, and pass it on every server. That fails because each backend has limits that a benchmark does not show.

trtllm_mha rejects speculative decoding with topk above 1 and does not run on SM120. FlashInfer cannot run models that use attention sinks. fa4 has a native page size of 128. Some backends accept FP8 KV caches and some do not. A kernel that wins on one model and GPU can refuse to start on the next.

How SGLang picks a default

When you do not pass --attention-backend, startup calls one function that reads your GPU, the attention type of the model, and your speculative decoding settings.

Fig. 4.3the default attention backend at v0.5.22decision

Original SVG

Original source

id: fig-4-3
kind: decision
title: the default attention backend at v0.5.22
claim: SGLang picks the attention backend from the GPU generation and the attention type of the model, and each branch records a measured limit.
caption: Read top to bottom. MHA models follow the left path and MLA models the right. Hopper means CUDA 12.3 or later. From get_default_attn_backend at v0.5.22.

For MHA models, such as Llama and Qwen, the order is this:

1. On Hopper with CUDA 12.3 or later, and no speculative topk above 1, the default is fa3. The code comment cites issue #17411.
2. On Blackwell SM100 or SM103, the default is trtllm_mha, or fa4 when K and V widths differ.
3. On AMD the default is aiter, and on Apple MPS it is torch_native.
4. In every other case the default is flashinfer when it is installed and the model has no attention sinks, else triton.

For MLA models, such as DeepSeek V3, Hopper gets fa3 and Blackwell gets flashinfer. AMD gets aiter when each rank has 16 or 128 KV heads. A model override then moves DeepSeek V3 on SM100 to trtllm_mla.

The docstring of the same function says FA3 is avoided when the page size is above 1. The code does not check the page size. When a comment and the code disagree, the code is what runs.

Two backends in one server

Prefill and decode stress different things. Prefill reads long query blocks, and decode reads one query against a long cache. --prefill-attention-backend and --decode-attention-backend set each phase separately. When they differ, SGLang wraps both in a hybrid backend that dispatches by ForwardMode. With speculative decoding, --speculative-attention-mode decides which of the two runs the verify pass. Its default is prefill.

Docs and code disagree

The attention backend page in the docs at v0.5.22 still lists Cutlass MLA and Dual Chunk FlashAttention. PR #32114 deleted both on 2026-09-10, and neither name is a valid choice at the tag. The same page says Blackwell MHA uses trtllm_mha. The code picks fa4 for models with asymmetric K and V widths. In both cases the code wins, and this manual follows the code.

What to measure

Read the resolved backend in the startup log before you benchmark anything. Then change one flag, rerun the same benchmark at your real concurrency, and compare inter-token latency at p50 and p99. Attention is one part of a decode step, so a faster kernel can move end-to-end numbers less than its own benchmark suggests.

On our Mac capture, SGLang ran through the MLX runtime. The server arguments resolve to torch_native, but MLX runs its own attention path, so none of these backends ran there.

What we measured on an L4

We ran the load sweep of section 5.1 on one L4 with Qwen/Qwen3-8B, three seeds per point. The default server resolved its backend to flashinfer and ran on 2026-10-16. The second server set --attention-backend triton and ran on 2026-10-18. Both had the same pool of 24,094 tokens and replayed decode graphs up to 48 running requests. Every run had a median SM clock of 1,000 MHz or more.

[2026-10-18 14:28:40] KV Cache is allocated. dtype: torch.bfloat16, #tokens: 24094, K size: 1.65 GB, V size: 1.65 GB
[2026-10-18 16:11:42] Decode batch, #running-req: 33, #token: 20422, token usage: 0.85, cuda graph: True, gen throughput (token/s): 311.50, #queue-req: 0

title: the Triton server under load
source: capture/gpu/out-l4/server-load-triton.log
lang: text
note: The pool at startup, and one decode line at 33 running requests.
---
[2026-10-18 14:28:40] KV Cache is allocated. dtype: torch.bfloat16, #tokens: 24094, K size: 1.65 GB, V size: 1.65 GB
[2026-10-18 16:11:42] Decode batch, #running-req: 33, #token: 20422, token usage: 0.85, cuda graph: True, gen throughput (token/s): 311.50, #queue-req: 0

We measured a tie on throughput and time per token. Throughput was within 4 percent and the median time per token within 1 ms at every concurrency. The median TTFT differed by up to 9 percent, in both directions, outside the seed spread at 2, 16, and 32 users. The two servers ran two days apart. Our reading: on a dense 8B model with contexts under 1,000 tokens, the attention kernel is a small part of each step. The weight reads set the pace.

Our position

Leave --attention-backend unset unless you have a benchmark on your own traffic. The default encodes regressions that other users found, such as #17411, and model overrides, such as DeepSeek V3 on SM100. The same issue is also the exception: a user who measures can beat a default that a library upgrade broke. Measure, then override, and write the reason next to the flag.

Prefer the backend that supports every feature you need over the one that wins a microbenchmark. Speculative decoding with topk above 1 needs flashinfer, fa3, or triton when the page size is above 1.

Incidents

A library upgrade made Hopper seven times slower. Issue #17411 (2026-01-20): FlashInfer 0.6.1 regressed the Hopper attention kernel. Median inter-token latency on H20 went from 39.44 ms to 283.52 ms. SGLang made fa3 the Hopper MHA default two days later.

The other engine lost its graphs. A field report by Varad More {{field Varad More 2026-04}} measured Gemma 3 4B on one A10G at 149 tok/s on SGLang and 84 tok/s on vLLM. The cause was outside attention speed. vLLM ran that model with --enforce-eager, without CUDA graphs, because of its interleaved sliding-window attention. A backend comparison is only fair when both servers replay graphs.

- Read the resolved attention backend in the startup log before every benchmark.
- Leave --attention-backend unset until a benchmark on your traffic shows a better choice.
- Check feature limits, such as speculative topk and page size, before choosing a backend.
- Pin your kernel library versions and rerun latency tests after every upgrade.

Sources: python/sglang/srt/arg_groups/model_override_base.py (get_default_attn_backend), arg_groups/choices.py (ATTENTION_BACKEND_CHOICES), layers/attention/base_attn_backend.py, layers/attention/attention_registry.py, model_executor/model_runner_components/attention_backend_setup.py, arg_groups/fields/spec.py, and docs/docs/advanced_features/attention_backend.mdx at v0.5.22; PR #32114; issue #17411; field report by Varad More (2026-04-22); capture/gpu/out-l4/server-load-triton.log, capture/gpu/summary/l4.json (load.default, load.triton).