self-hosted engine decision tree — hardware first, then scale, then workload
CPU / edge
llama.cpp (dominant)
· widest model support
· fastest CPU inference
· full control over quant / threads
Ollama = llama.cpp + UX
one command, single user
15-30% slower than llama.cpp
NVIDIA GPU
vLLM (broad default)
v0.15.1 Feb 2026, Blackwell SM120
TRT-LLM (Blackwell only)
throughput leader on B200/GB200
Ollama (dev only)
not for shared production
agentic / prefix-heavy
SGLang (specialist)
· RadixAttention (Phase 17 · 06)
· 400,000+ GPUs in production
· xAI, LinkedIn, Cursor, Oracle
· GCP, Azure, AWS
RAG heavy prefix reuse → here
multi-turn agent tools → here
TGI maintenance mode since December 11, 2025
only bug fixes going forward — do NOT default to TGI for new projects
existing TGI deployments can continue; migrate to SGLang or vLLM when convenient
2026 pipeline pattern
dev: Ollama → staging: llama.cpp → prod: vLLM (or SGLang)
same GGUF or HF weights across all three tiers
engineers iterate fast locally; staging mirrors prod quant; prod is the serving target