Intelligent LLM Autoscaling with KEDA and vLLM on OpenShift: Scaling on KV-Cache Pressure and ITL Latency

In enterprise large language model (LLM) production deployments, infrastructure autoscaling presents a critical operational and financial dilemma: On one hand, over-provisioning high-end GPU nodes (NVIDIA H100/H200, B200, or L40S) leads to astronomical cloud costs during off-peak hours. On the other hand, under-provisioning leads to queue saturation, memory exhaustion, and unacceptable spikes in Time-To-First-Token (TTFT) and Inter-Token Latency (ITL) for interactive users and AI agents.

The traditional reaction of infrastructure teams is to configure a standard Kubernetes Horizontal Pod Autoscaler (HPA) based on CPU utilization, system RAM, or generic GPU memory. However, for modern continuous-batching inference engines like vLLM or TGI, this approach is fundamentally broken: VRAM is pre-allocated up to 90% at engine boot via PagedAttention, leaving external memory metrics flat, static, and completely blind to real workload demands.

Learn how to implement intelligent event-driven autoscaling using KEDA (Kubernetes Event-driven Autoscaling) on Red Hat OpenShift AI (RHOAI), driving the lifecycle of inference replicas directly from internal KV-Cache pressure and real-time request queue metrics.

---

1. Why Traditional Autoscaling Fails for LLMs

To understand why traditional autoscaling mechanisms fail on GenAI workloads, we must analyze the internal mechanics of modern inference engines:

┌─────────────────────────────────────────────────────────────────────────────┐
│                       Why Standard HPA is Blind to LLMs                     │
├─────────────────────────────────────────────────────────────────────────────┤
│ 1. VRAM Memory : vLLM pre-allocates KV-Cache at boot (gpu_memory_util=0.90).│
│    => Kubernetes perceives 90% VRAM constantly (zero workload signal).      │
│                                                                             │
│ 2. GPU Utilization (%) : Wildly oscillates between Prefill and Decode.      │
│    => Causes rapid flapping, thrashing, and destructive pod termination.     │
│                                                                             │
│ 3. HTTP Request Rate (RPS) : Not all prompts are created equal!              │
│    => A 4,000-token prompt saturates the KV-Cache 20x more than short queries.│
└─────────────────────────────────────────────────────────────────────────────┘

To enable predictive, smooth, and cost-effective scaling, Kubernetes must monitor the internal state of the continuous batching iteration loop via metrics exposed by the vLLM /metrics endpoint.

---

2. Key vLLM Metrics for Production Autoscaling

vLLM natively exposes high-fidelity Prometheus metrics. Three core signals form the foundation of intelligent autoscaling:

1. vllm:num_requests_waiting : The count of incoming requests queued in vLLM’s internal scheduling queue waiting for available KV-Cache blocks. Any value above zero indicates immediate compute saturation.
2. vllm:gpu_cache_usage_factor : The proportion of KV-Cache memory blocks currently allocated for active decoding sequences (normalized from 0.0 to 1.0).
3. vllm:time_to_first_token_seconds_bucket / vllm:time_per_output_token_seconds : The effective end-to-end token latency experienced by clients, ideal for alerting and SLA guardrails.

                  ┌─────────────────────────────────────────────────────────┐
                  │       Ingress Traffic (Incoming LLM Requests)           │
                  └────────────────────────────┬────────────────────────────┘
                                               │
                                               ▼
                  ┌─────────────────────────────────────────────────────────┐
                  │     OpenShift AI / KServe InferenceService (vLLM)       │
                  │   ┌─────────────────────────────────────────────────┐   │
                  │   │ Metrics Exporter: /metrics                      │   │
                  │   │ • vllm:gpu_cache_usage_factor                   │   │
                  │   │ • vllm:num_requests_waiting                     │   │
                  │   └────────────────────────┬────────────────────────┘   │
                  └────────────────────────────┼────────────────────────────┘
                                               │ Scrape (10s)
                                               ▼
                  ┌─────────────────────────────────────────────────────────┐
                  │             OpenShift User-Workload Monitoring          │
                  │                   (Prometheus / Thanos)                 │
                  └────────────────────────────┬────────────────────────────┘
                                               │ Query PromQL
                                               ▼
                  ┌─────────────────────────────────────────────────────────┐
                  │                 KEDA Autoscaling Operator               │
                  │ (ScaledObject: triggers on cache > 0.85 or waiting > 2) │
                  └────────────────────────────┬────────────────────────────┘
                                               │ Scale Out / Scale In
                                               ▼
                  ┌─────────────────────────────────────────────────────────┐
                  │      OpenShift GPU Orchestrator (DRA / MachineSet)      │
                  │      Spawns new vLLM Pods on available GPU Worker Nodes │
                  └─────────────────────────────────────────────────────────┘

---

3. Declarative Architecture: KEDA & ScaledObject Configuration

On Red Hat OpenShift, KEDA is deployed directly via OperatorHub and natively integrates with OpenShift User-Workload Monitoring to query Prometheus metrics securely.

Step 1: Enable the ServiceMonitor for vLLM

This manifest instructs OpenShift Monitoring to scrape the vLLM runtime metrics every 10 seconds:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: vllm-metrics-monitor
  namespace: ai-inference-prod
  labels:
    app.kubernetes.io/name: vllm-granite-3b
spec:
  selector:
    matchLabels:
      serving.kserve.io/inferenceservice: granite-3b-instruct
  endpoints:
    - port: metrics
      path: /metrics
      interval: 10s
      scrapeTimeout: 5s

Step 2: Define KEDA TriggerAuthentication for Thanos Querier

To securely allow KEDA to query OpenShift’s internal Thanos metrics endpoint:

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-trigger-auth-prometheus
  namespace: ai-inference-prod
spec:
  secretTargetRef:
    - parameter: bearerToken
      name: thanos-querier-token
      key: token
    - parameter: ca
      name: thanos-querier-ca
      key: ca.crt

Step 3: Deploy the Multi-Trigger KEDA ScaledObject

The following ScaledObject dynamically adjusts the vLLM replica count between 1 and 8 pods based on dual criteria:

• If vllm:num_requests_waiting exceeds 2 pending requests.
• If the average vllm:gpu_cache_usage_factor exceeds 0.80 (80% KV-Cache utilization).

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-granite-scaledobject
  namespace: ai-inference-prod
spec:
  scaleTargetRef:
    apiVersion: serving.kserve.io/v1beta1
    kind: InferenceService
    name: granite-3b-instruct
  minReplicaCount: 1
  maxReplicaCount: 8
  cooldownPeriod: 300   # 5-minute cooldown prevents rapid scale-in thrashing
  pollingInterval: 15  # Evaluation interval
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 0
          policies:
            - type: Percent
              value: 100
              periodSeconds: 15
        scaleDown:
          stabilizationWindowSeconds: 300
          policies:
            - type: Pods
              value: 1
              periodSeconds: 60
  triggers:
    - type: prometheus
      metadata:
        serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091
        metricName: vllm_waiting_requests_trigger
        threshold: '2'
        query: sum(vllm:num_requests_waiting{namespace="ai-inference-prod", model_name="granite-3b-instruct"})
        authModes: bearer
      authenticationRef:
        name: keda-trigger-auth-prometheus

    - type: prometheus
      metadata:
        serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091
        metricName: vllm_kv_cache_pressure_trigger
        threshold: '0.80'
        query: avg(vllm:gpu_cache_usage_factor{namespace="ai-inference-prod", model_name="granite-3b-instruct"})
        authModes: bearer
      authenticationRef:
        name: keda-trigger-auth-prometheus

---

4. Overcoming the GPU “Cold Start” Challenge on OpenShift

While KEDA triggers pod scaling in milliseconds, the actual spin-up time for a vLLM pod (Cold Start) can range between 45 to 120 seconds due to:

1. Dynamic GPU node provisioning via OpenShift MachineSets / Cluster Autoscaler.
2. Model weight transfer (10 GB to 70 GB) from S3/Ceph object storage.
3. CUDA kernel compilation and PyTorch execution graph initialization.

Engineering Best Practices to Mitigate Cold Starts:

• Shared ReadWriteMany Model Cache (ODF / CephFS / Lustre): Mount a shared read-only cache volume so newly spawned pods instantly access model weights without downloading over the network.
• Warm Pools & Pre-warmed Worker Nodes: Maintain at least one active replica (minReplicaCount: 1) and keep pre-pulled container images across GPU nodes.
• Conservative Scale-In Cooldown: Set a cooldownPeriod of at least 300 seconds (5 minutes) to avoid terminating recently warmed GPU pods right before a follow-up burst.

---

5. Autoscaling Strategies Comparison for Production LLMs

Evaluation Criteria	Standard HPA (CPU / RAM)	Custom HPA (GPU Memory Util)	KEDA + vLLM Metrics (KV-Cache & Queue)
Signal Accuracy	Zero (completely uncorrelated)	Misleading (VRAM pre-allocated at 90%)	Maximal (real continuous batching pressure)
Burst Reactivity	Extremely slow	Ineffective	Instantaneous (triggers on requests_waiting > 2)
Long Context Handling	Blind	Blind	Native (direct tracking of KV-Cache allocation)
GPU Cost Optimization	Poor (high OOM crash risk)	Poor (permanent over-provisioning)	Optimal (surgical scale-out and controlled scale-in)
OpenShift Integration	Basic	Requires NVIDIA GPU Operator	Seamless via KEDA Operator + OpenShift Monitoring

---

6. Platform & LLMOps Implementation Checklist

To successfully industrialize LLM autoscaling across your enterprise clusters:

1. Always enable vLLM Prometheus metrics: Ensure --disable-log-stats is disabled and the metrics port is routed to your ServiceMonitor.
2. Combine dual triggers: Pair memory pressure (gpu_cache_usage_factor) for long context prompts with queue depth (num_requests_waiting) for concurrent user surges.
3. Monitor Time-To-First-Token (TTFT): If median TTFT breaches 800 ms, tighten the trigger thresholds to scale out more aggressively.
4. Enforce Declarative GitOps (ArgoCD): Store all ScaledObject, ServiceMonitor, and TriggerAuthentication manifests in Git for reproducible multi-cluster deployments.

---

Master Production AI Systems Engineering with Aperta Scientia

Deploying and scaling enterprise foundation models at scale requires dual mastery: expert Cloud-Native Kubernetes orchestration and deep optimization of GenAI inference engines.

Aperta Scientia prepares your engineering teams for these cutting-edge challenges:

• AS300 — AI Platform Engineer: The elite 3-month curriculum to master OpenShift AI (RHOAI), vLLM optimization, event-driven scaling with KEDA, declarative serving with KServe, vector databases with Milvus, and governance with TrustyAI.
• AS200 — New DevOps Platform Engineer : The foundational curriculum to design resilient Red Hat OpenShift clusters, automate with GitOps (ArgoCD), implement robust Prometheus / Thanos observability, and manage multi-cluster environments with RHACM.