AI gateways — seven features, four 2026 leaders
core features
· provider routing
· fallback chains
· retries (backoff)
· rate limits (per-tenant)
· secret references
· observability (OTel)
· guardrails (PII, jailbreak)
all seven mandatory at enterprise scale
2026 gateways
LiteLLM — MIT, Python
Portkey — Apache 2.0, guardrails
Kong AI Gateway — enterprise
Cloudflare / Vercel — edge managed
scale + overhead (Kong benchmark)
Kong: ~5 ms overhead (baseline)
LiteLLM: ~10 ms overhead (breaks >2K RPS)
Portkey: ~30 ms overhead
Cloudflare: ~2 ms (edge)
Kong 228% faster than Portkey
Kong 859% faster than LiteLLM
on same 12 CPUs
LiteLLM for dev, Kong for prod scale
data residency drives self-host
healthcare / finance → self-hosted
LiteLLM OSS, Portkey OSS, Kong OSS
consumer → managed edge
Cloudflare AI Gateway, Vercel AI
Portkey Apache 2.0 March 2026 moved middle tier
compose with Phase 17 · 13 + 16
gateway (19) + observability (13) + routing (16)
often the same tool:
Portkey = gateway + observability + guardrails
Helicone = gateway + observability
pick one, or wire three with OpenTelemetry
fallback arithmetic — OpenAI 3% × Anthropic 2% = 0.06% both-fail
three-provider chain effectively 99.997% availability