# Intelligent LLM Autoscaling with KEDA and vLLM on OpenShift: Scaling on KV-Cache Pressure and ITL Latency > Comprehensive engineering guide for dynamically scaling production LLM servers: why standard HPA fails and how to configure KEDA with native vLLM metrics. - **Date de publication** : 2026-09-14T08:00:00.000Z --- In enterprise large language model (LLM) production deployments, **infrastructure autoscaling** presents a critical operational and financial dilemma: On one hand, over-provisioning high-end GPU nodes (NVIDIA H100/H200, B200, or L40S) leads to astronomical cloud costs during off-peak hours. On the other hand, under-provisioning leads to queue saturation, memory exhaustion, and unacceptable spikes in **Time-To-First-Token (TTFT)** and **Inter-Token Latency (ITL)** for interactive users and AI agents. The traditional reaction of infrastructure teams is to configure a standard Kubernetes **Horizontal Pod Autoscaler (HPA)** based on CPU utilization, system RAM, or generic GPU memory. However, for modern continuous-batching inference engines like **vLLM** or **TGI**, this approach is fundamentally broken: VRAM is pre-allocated up to 90% at engine boot via **PagedAttention**, leaving external memory metrics flat, static, and completely blind to real workload demands. Learn how to implement **intelligent event-driven autoscaling using KEDA (Kubernetes Event-driven Autoscaling)** on **Red Hat OpenShift AI (RHOAI)**, driving the lifecycle of inference replicas directly from internal **KV-Cache pressure** and real-time request queue metrics. --- ### 1. Why Traditional Autoscaling Fails for LLMs To understand why traditional autoscaling mechanisms fail on GenAI workloads, we must analyze the internal mechanics of modern inference engines: ``` ┌─────────────────────────────────────────────────────────────────────────────┐ │ Why Standard HPA is Blind to LLMs │ ├─────────────────────────────────────────────────────────────────────────────┤ │ 1. VRAM Memory : vLLM pre-allocates KV-Cache at boot (gpu_memory_util=0.90).│ │ => Kubernetes perceives 90% VRAM constantly (zero workload signal). │ │ │ │ 2. GPU Utilization (%) : Wildly oscillates between Prefill and Decode. │ │ => Causes rapid flapping, thrashing, and destructive pod termination. │ │ │ │ 3. HTTP Request Rate (RPS) : Not all prompts are created equal! │ │ => A 4,000-token prompt saturates the KV-Cache 20x more than short queries.│ └─────────────────────────────────────────────────────────────────────────────┘ ``` To enable predictive, smooth, and cost-effective scaling, Kubernetes must monitor the internal state of the continuous batching iteration loop via metrics exposed by the vLLM `/metrics` endpoint. --- ### 2. Key vLLM Metrics for Production Autoscaling vLLM natively exposes high-fidelity Prometheus metrics. Three core signals form the foundation of intelligent autoscaling: 1. **`vllm:num_requests_waiting` :** The count of incoming requests queued in vLLM's internal scheduling queue waiting for available KV-Cache blocks. Any value above zero indicates immediate compute saturation. 2. **`vllm:gpu_cache_usage_factor` :** The proportion of KV-Cache memory blocks currently allocated for active decoding sequences (normalized from `0.0` to `1.0`). 3. **`vllm:time_to_first_token_seconds_bucket` / `vllm:time_per_output_token_seconds` :** The effective end-to-end token latency experienced by clients, ideal for alerting and SLA guardrails. ``` ┌─────────────────────────────────────────────────────────┐ │ Ingress Traffic (Incoming LLM Requests) │ └────────────────────────────┬────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────┐ │ OpenShift AI / KServe InferenceService (vLLM) │ │ ┌─────────────────────────────────────────────────┐ │ │ │ Metrics Exporter: /metrics │ │ │ │ • vllm:gpu_cache_usage_factor │ │ │ │ • vllm:num_requests_waiting │ │ │ └────────────────────────┬────────────────────────┘ │ └────────────────────────────┼────────────────────────────┘ │ Scrape (10s) ▼ ┌─────────────────────────────────────────────────────────┐ │ OpenShift User-Workload Monitoring │ │ (Prometheus / Thanos) │ └────────────────────────────┬────────────────────────────┘ │ Query PromQL ▼ ┌─────────────────────────────────────────────────────────┐ │ KEDA Autoscaling Operator │ │ (ScaledObject: triggers on cache > 0.85 or waiting > 2) │ └────────────────────────────┬────────────────────────────┘ │ Scale Out / Scale In ▼ ┌─────────────────────────────────────────────────────────┐ │ OpenShift GPU Orchestrator (DRA / MachineSet) │ │ Spawns new vLLM Pods on available GPU Worker Nodes │ └─────────────────────────────────────────────────────────┘ ``` --- ### 3. Declarative Architecture: KEDA & ScaledObject Configuration On Red Hat OpenShift, KEDA is deployed directly via OperatorHub and natively integrates with **OpenShift User-Workload Monitoring** to query Prometheus metrics securely. #### Step 1: Enable the ServiceMonitor for vLLM This manifest instructs OpenShift Monitoring to scrape the vLLM runtime metrics every 10 seconds: ```yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: vllm-metrics-monitor namespace: ai-inference-prod labels: app.kubernetes.io/name: vllm-granite-3b spec: selector: matchLabels: serving.kserve.io/inferenceservice: granite-3b-instruct endpoints: - port: metrics path: /metrics interval: 10s scrapeTimeout: 5s ``` #### Step 2: Define KEDA TriggerAuthentication for Thanos Querier To securely allow KEDA to query OpenShift's internal Thanos metrics endpoint: ```yaml apiVersion: keda.sh/v1alpha1 kind: TriggerAuthentication metadata: name: keda-trigger-auth-prometheus namespace: ai-inference-prod spec: secretTargetRef: - parameter: bearerToken name: thanos-querier-token key: token - parameter: ca name: thanos-querier-ca key: ca.crt ``` #### Step 3: Deploy the Multi-Trigger KEDA ScaledObject The following `ScaledObject` dynamically adjusts the vLLM replica count between 1 and 8 pods based on dual criteria: * If **`vllm:num_requests_waiting`** exceeds **2** pending requests. * If the average **`vllm:gpu_cache_usage_factor`** exceeds **0.80** (80% KV-Cache utilization). ```yaml apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: vllm-granite-scaledobject namespace: ai-inference-prod spec: scaleTargetRef: apiVersion: serving.kserve.io/v1beta1 kind: InferenceService name: granite-3b-instruct minReplicaCount: 1 maxReplicaCount: 8 cooldownPeriod: 300 # 5-minute cooldown prevents rapid scale-in thrashing pollingInterval: 15 # Evaluation interval advanced: horizontalPodAutoscalerConfig: behavior: scaleUp: stabilizationWindowSeconds: 0 policies: - type: Percent value: 100 periodSeconds: 15 scaleDown: stabilizationWindowSeconds: 300 policies: - type: Pods value: 1 periodSeconds: 60 triggers: - type: prometheus metadata: serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091 metricName: vllm_waiting_requests_trigger threshold: '2' query: sum(vllm:num_requests_waiting{namespace="ai-inference-prod", model_name="granite-3b-instruct"}) authModes: bearer authenticationRef: name: keda-trigger-auth-prometheus - type: prometheus metadata: serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091 metricName: vllm_kv_cache_pressure_trigger threshold: '0.80' query: avg(vllm:gpu_cache_usage_factor{namespace="ai-inference-prod", model_name="granite-3b-instruct"}) authModes: bearer authenticationRef: name: keda-trigger-auth-prometheus ``` --- ### 4. Overcoming the GPU "Cold Start" Challenge on OpenShift While KEDA triggers pod scaling in milliseconds, the actual spin-up time for a vLLM pod (*Cold Start*) can range between 45 to 120 seconds due to: 1. Dynamic GPU node provisioning via OpenShift MachineSets / Cluster Autoscaler. 2. Model weight transfer (10 GB to 70 GB) from S3/Ceph object storage. 3. CUDA kernel compilation and PyTorch execution graph initialization. #### Engineering Best Practices to Mitigate Cold Starts: * **Shared ReadWriteMany Model Cache (ODF / CephFS / Lustre):** Mount a shared read-only cache volume so newly spawned pods instantly access model weights without downloading over the network. * **Warm Pools & Pre-warmed Worker Nodes:** Maintain at least one active replica (`minReplicaCount: 1`) and keep pre-pulled container images across GPU nodes. * **Conservative Scale-In Cooldown:** Set a `cooldownPeriod` of at least 300 seconds (5 minutes) to avoid terminating recently warmed GPU pods right before a follow-up burst. --- ### 5. Autoscaling Strategies Comparison for Production LLMs | Evaluation Criteria | Standard HPA (CPU / RAM) | Custom HPA (GPU Memory Util) | KEDA + vLLM Metrics (KV-Cache & Queue) | | :--- | :--- | :--- | :--- | | **Signal Accuracy** | Zero (completely uncorrelated) | Misleading (VRAM pre-allocated at 90%) | **Maximal (real continuous batching pressure)** | | **Burst Reactivity** | Extremely slow | Ineffective | **Instantaneous (triggers on `requests_waiting > 2`)** | | **Long Context Handling** | Blind | Blind | **Native (direct tracking of KV-Cache allocation)** | | **GPU Cost Optimization** | Poor (high OOM crash risk) | Poor (permanent over-provisioning) | **Optimal (surgical scale-out and controlled scale-in)** | | **OpenShift Integration** | Basic | Requires NVIDIA GPU Operator | **Seamless via KEDA Operator + OpenShift Monitoring** | --- ### 6. Platform & LLMOps Implementation Checklist To successfully industrialize LLM autoscaling across your enterprise clusters: 1. **Always enable vLLM Prometheus metrics:** Ensure `--disable-log-stats` is disabled and the metrics port is routed to your ServiceMonitor. 2. **Combine dual triggers:** Pair memory pressure (`gpu_cache_usage_factor`) for long context prompts with queue depth (`num_requests_waiting`) for concurrent user surges. 3. **Monitor Time-To-First-Token (TTFT):** If median TTFT breaches 800 ms, tighten the trigger thresholds to scale out more aggressively. 4. **Enforce Declarative GitOps (ArgoCD):** Store all `ScaledObject`, `ServiceMonitor`, and `TriggerAuthentication` manifests in Git for reproducible multi-cluster deployments. --- ### Master Production AI Systems Engineering with Aperta Scientia Deploying and scaling enterprise foundation models at scale requires dual mastery: expert Cloud-Native Kubernetes orchestration and deep optimization of GenAI inference engines. Aperta Scientia prepares your engineering teams for these cutting-edge challenges: * **[AS300 — AI Platform Engineer](/curriculums/as300)**: The elite 3-month curriculum to master **OpenShift AI (RHOAI)**, **vLLM** optimization, event-driven scaling with **KEDA**, declarative serving with **KServe**, vector databases with **Milvus**, and governance with **TrustyAI**. * **[AS200 — New DevOps Platform Engineer](/curriculums/as200)** : The foundational curriculum to design resilient **Red Hat OpenShift** clusters, automate with **GitOps (ArgoCD)**, implement robust **Prometheus / Thanos** observability, and manage multi-cluster environments with **RHACM**. --- *Source : https://aperta.io/radar/2026-09-14-autoscaling-llm-keda-vllm-kv-cache-openshift/ — Aperta Scientia Tech Radar*