RAG evaluation — four dimensions, LLM-as-judge backend
question: "When was iPhone released?"
retrieved context:
"Apple released first iPhone
June 29, 2007."
answer: "June 29, 2007."
LLM judge
(GPT-4o-mini, Haiku, Mistral-small)
score(q, ctx, ans) → 0..1
rubric + chain-of-thought
calibrated vs human
four RAG metrics
faithfulness
claims in answer entailed by context?
answer relevance
answer addresses the question?
context precision
fraction of retrieved chunks relevant
context recall
retrieved set covers gold answer?
cost / calibration / CI
$0.001 – $0.003 per judged case with GPT-4o-mini — <$5 per 1000-sample regression run
required: Spearman ρ > 0.7 vs 50 hand-labeled cases before trusting the number
gate: pytest + DeepEval thresholds; freeze judge model + version; surface bottom-10%