LLM routing gateway: one API, N providers
your app
openai.chat.completions.create(
model="smart",
messages=[...])
- alias "smart" is a logical
name, not a real model
- no provider-specific code
routing gateway
1 / redact PII
regex or ML pre-dispatch
2 / semantic cache
prompt embedding lookup
3 / alias resolve
smart -> [gpt-4o, claude, gemini]
4 / pick strategy
static / cost / latency / A-B
5 / call provider
translate OpenAI -> backend shape
6 / fallback on 5xx
next provider in chain
7 / cost tracking
tokens x price -> per-key aggregate
8 / emit OTel GenAI
gen_ai.* on every hop
9 / guardrails on output
scrub completions before return
providers
OpenAI
gpt-4o, gpt-4o-mini
Anthropic
claude-sonnet, haiku
Google
gemini-pro, flash
Cohere / Mistral /
DeepSeek / Ollama
choices (2026)
LiteLLM (self-host)
OpenRouter (SaaS)
Portkey (prod)
key trade-off
control vs managed
vs guardrails out
of the box