A/B testing LLM features — evals + online, not vibes
evals (offline, labeled)
"can the model do the job?"
· human-labeled or LLM-as-judge
· fixed distribution (not user traffic)
· catches regressions before exposure
A/B tests (online, random)
"do users care?"
· live traffic, randomized assignment
· user-level metric that matters (retention, CTR)
· confirms product impact
what to test
· prompt wording
· model selection (GPT vs OSS)
· generation params (temp, top-p)
· reward functions / routers
method rigor
· CUPED (30-70% variance cut)
· sequential (always-valid peeks)
· Bonferroni / Benjamini-Hochberg
· SRM check (sample ratio)
platforms
Statsig (OpenAI-owned, $1.1B)
GrowthBook (MIT, warehouse-native)
Eppo, Optimizely Experiments
LaunchDarkly Experiments
LLM non-determinism complicates power
up to 15% run-to-run variance on identical inputs
buffer sample size ×1.3-1.5 beyond traditional power calc
real cases: chatbot +70% conv length · Nextdoor +1% CTR · Khanmigo latency-vs-accuracy loop
evals catch regressions; A/B confirms impact; shipping on vibes is over