self-hosted LLM observability plane — OTel GenAI semconv
instrumented apps
OpenAI SDK Anthropic SDK
Google GenAI LangChain
LlamaIndex vLLM
OpenLLMetry auto-instrument
OTLP HTTP -> collector
six SDK families covered
collector
OTLP HTTP receiver
tail-sampling processor
keep all errors
keep toxicity / PII spikes
10% stratified success sample
exporter: ClickHouse + S3
storage
ClickHouse: spans (columnar)
Postgres: metadata
S3: raw event archive
schema mirrors GenAI semconv
gen_ai.system, request.model
usage.input_tokens, output_tokens
evals + drift (child spans)
DeepEval / RAGAS scheduled jobs
custom LLM-judges (PII, off-policy)
drift detector (weekly PSI / KL)
eval outputs become linked child spans
stored in same ClickHouse, join by parent
tail-sampler keeps all high-eval-score traces
MTTR on injected regression target: < 5 min
alerting + UI
Prometheus Alertmanager
Slack (warning) + PagerDuty (critical)
Next.js 15 + Recharts dashboard
pages: overview / traces / evals / drift
cost per user, span waterfall, eval trends
Langfuse / Phoenix / OpenLLMetry shape
open-core self-hosted reference