speculative decoding — draft proposes, target verifies
draft model (cheap)
EAGLE-3 head
P-EAGLE parallel tree
ngram fallback
proposes k tokens per step
trained on target hidden states
via SpecForge / Speculators
alignment = acceptance knob
target model (expensive)
batched verify pass
one forward through k positions
accept prefix
until first rejection
resample rejected position from target
commit accepted + 1 new -> stream
target quantization
FP8 Marlin or INT4 AWQ
Llama 3.3 70B, Qwen3-Coder-30B MoE, GPT-OSS-120B
one replica per H100/H200 shard
vLLM 0.7 / SGLang 0.4
metrics that must be reported
throughput bs=1 / bs=8 / bs=32 with + without spec
p50, p99 latency at each batch size
acceptance rate per distribution:
ShareGPT, HumanEval, domain data
draft/target alignment drift alert
$ per 1M output tokens sustained
comparison vs Anthropic / OpenAI API on same eval
operations
Kubernetes deployment
NVIDIA device plugin
HPA metric: queue_wait_ms (not CPU)
rolling upgrade: drain queue first
ngram fallback if EAGLE acceptance drops
Prometheus exporter + Langfuse trace
target: 2.5x+ throughput at matched quality