LLM rollout sequence — shadow → canary → A/B → 100%
1. shadow mode
zero user impact
· duplicate prod requests to candidate
· log outputs, token counts, latency
· diff vs production output
· catches: cost spikes, length shifts,
obvious refusal changes, hard errors
not a quality test — a smoke test
2. canary rollout
1% → 10% → 25% → 50% → 75% → 100%
· five gates at each step:
latency P99 > 1.5x baseline
cost/request > 1.2x baseline
error/refusal > 2x baseline
output-length shift > 1.4x
thumbs-down > 1.5x baseline
3. A/B (optional)
only for distinct alternatives
· 50/50 split
· run until stats significance
· CUPED / sequential / Benjamini-H
· skip if just improved variant
· Phase 17 · 21 covers GrowthBook +
Statsig semantics
non-determinism sets the noise floor
up to 15% run-to-run variance on identical inputs
causes: GPU FP non-associativity, batch-size variance, sampling
gates must sit above the noise floor, not at identity with baseline
rollback in seconds
policy flag (feature flags) + model pin (registry digest)
rollback = flip flag + revert digest
if rollback requires redeploy you are too slow — fix the stack first
tooling: Argo Rollouts, Flagger, Istio weighted, KServe, feature flag system