# GenAI Observability: Standardizing LLM Serving Telemetry with OpenTelemetry

> How OpenTelemetry GenAI semantic conventions standardize TTFT, inter-token latency, and KV-cache monitoring across vLLM and KServe.

- **Date de publication** : 2026-09-08T08:30:00.000Z

---

Operating LLM ServingRuntimes (**vLLM**, **KServe**) on Kubernetes and OpenShift clusters without standardized telemetry leads to blind spots during traffic spikes.

The stabilization of **OpenTelemetry Semantic Conventions for Generative AI** establishes a vendor-neutral standard to monitor, trace, and optimize model serving performance in production.

---

### 1. The Limits of Traditional Application Metrics on LLMs
For standard HTTP services, overall request latency is typically sufficient. In streaming LLM architectures, aggregate duration hides critical bottlenecks:
* **Time to First Token (TTFT):** Measures the prompt prefill duration and scheduling latency. TTFT spikes indicate GPU compute saturation or excessive queuing.
* **Inter-Token Latency (ITL / Time per Output Token):** Measures token-by-token decode speed, directly tied to GPU VRAM memory bandwidth.
* **Token Usage & Quota Tracking:** Granular accounting of prompt and completion tokens is essential for multi-tenant rate limiting and OOM prevention.

---

### 2. OpenTelemetry GenAI Standard in Production
Integrating these conventions into cloud-native observability pipelines (**Prometheus**, **Vector**, OpenShift Observability):
* **Standardized Metrics:** `gen_ai.server.time_to_first_token`, `gen_ai.server.inter_token_latency`, `gen_ai.client.token.usage`.
* **Unified Distributed Tracing:** Seamless span correlation across prompt prefill, vector database retrieval (RAG), and client application queries.
* **SLO-Driven Autoscaling:** Feeding OpenTelemetry SLIs into KEDA and Horizontal Pod Autoscalers to scale GPU workers proactively before queue saturation.

---

### 3. Production Skills in Aperta Scientia Curriculums
* **AS300 (AI Platform Engineer):** Hands-on instrumentation of vLLM and KServe runtimes, Grafana dashboarding, and correlation with bare-metal NVIDIA GPU metrics.
* **AS200 (DevOps Platform Engineer):** Full observability stack integration (Prometheus, Thanos, Grafana) and SLO-based automated alerting.

---

### 4. Technical Sources & References
* **OpenTelemetry GenAI Semantic Conventions:** [OpenTelemetry Official Specification](https://opentelemetry.io/docs/specs/semconv/gen-ai/)
* **vLLM Metrics & Prometheus Integration:** [vLLM Production Metrics](https://docs.vllm.ai/en/latest/serving/metrics.html)
* **KServe Observability & Telemetry:** [KServe Telemetry Guide](https://kserve.github.io/website/)

---

*Source : https://aperta.io/radar/2026-09-08-opentelemetry-genai-semantic-conventions-llm-observability/ — Aperta Scientia Tech Radar*