llm-d & Distributed Inference on OpenShift: Prefill / Decode Disaggregation and Cache-Aware Routing

Scaling enterprise generative AI inference encounters a fundamental architectural bottleneck: the mismatch between the compute profiles of the Prefill stage (prompt ingestion and Key-Value cache computation) and the Decode stage (sequential token generation). In standard monolithic vLLM deployments, both phases share the same GPU, leading to severe resource contention on long-context queries and unpredictable Inter-Token Latency (ITL) jitter.

The llm-d architecture—a CNCF Sandbox project backed by Red Hat, IBM Research, Google Cloud, CoreWeave, and NVIDIA—standardizes Kubernetes-native orchestration for disaggregated inference (Prefill/Decode Disaggregation) and KV-cache-aware routing on Red Hat OpenShift AI.

---

1. The Monolithic Bottleneck: Conflicting Compute Phases

LLM inference operates under two fundamentally different execution profiles:

1. Prefill Phase (Compute-Bound): Highly parallel processing of the input sequence to compute initial attention matrices and generate the KV cache. This phase saturates Tensor Cores and requires raw compute throughput (TFLOPS).
2. Decode Phase (Memory-Bandwidth-Bound): Autoregressive token-by-token generation. Each new token requires minimal matrix multiplication but heavily taxes High Bandwidth Memory (HBM) bandwidth to reload model weights and KV-cache tensors at every step.

In a monolithic setup, large input prompts (e.g., 64k or 128k tokens in enterprise RAG pipelines) monopolize GPU cores, stalling token generation for concurrent active streams. As a result, Time-To-First-Token (TTFT) spikes and Inter-Token Latency (ITL) degrades.

---

2. Prefill / Decode (P/D) Disaggregation with vLLM

Disaggregated inference splits the serving pipeline into two dedicated, independently scalable worker pools:

• Prefill Pool (Compute-Optimized Instances): Ingests incoming prompts, computes the KV cache at maximum throughput, and immediately offloads the tensor state without running the token decode loop.
• Decode Pool (Memory-Bandwidth-Optimized Instances): Ingests the pre-computed KV-cache tensor and streams output tokens sequentially with minimal, predictable latency.

Inter-node KV-cache transfer is handled via high-speed interfaces (vLLM NIXL / UCX / RDMA over Converged Ethernet), completely eliminating redundant prompt reprocessing on decode nodes.

---

3. The Role of llm-d: Intelligent Scheduling and Gateway API Inference Extension

llm-d delivers the Kubernetes-native control plane required to orchestrate disaggregated clusters:

• InferencePool and InferenceObjective CRDs: Declarative specification of specialized compute pools (NVIDIA H100/A100, AMD Instinct, L40S) and explicit SLO targets (target TTFT, p99 token latency).
• Prefix-Cache-Aware Routing: Operating via the Kubernetes Gateway API Inference Extension, the inference gateway maintains a distributed prefix index. Prompts sharing common prefixes (system instructions, recurring RAG documents) are automatically dispatched to pods that already hold cached KV blocks, cutting TTFT by up to 90%.
• Smart Load Balancing: Dynamically evaluates GPU memory pressure and real-time active batch sizes across vLLM pods rather than relying on naive round-robin scheduling.

---

4. Operationalization on Red Hat OpenShift AI

Deploying llm-d on Red Hat OpenShift leverages core enterprise platform primitives:

• KServe & ServingRuntime: Declarative management and lifecycle operations for containerized vLLM serving workloads.
• Independent Autoscaling with KEDA: Decoupled scaling of Prefill pods based on queue depth and token volume without artificially over-provisioning Decode capacity.
• High-Speed Networking (SR-IOV / RoCE): Deterministic high-bandwidth networking enabling sub-millisecond KV-cache handoffs across cluster nodes.
• OpenTelemetry & Prometheus Observability: Full visibility into golden inference metrics (vllm:time_to_first_token_seconds, vllm:time_per_output_token_seconds, cache hit ratios).

---

5. Curriculum Alignment & Skills at Aperta Scientia

• AS300 Curriculum (AI Platform Engineer — 399h): Core alignment with distributed AI inference architectures—advanced vLLM runtime configuration, KServe ServingRuntimes under Red Hat OpenShift AI (RHOAI), distributed KV-cache management, llm-d InferencePool CRDs, and inference performance benchmarking (GuideLLM / lm_eval).
• AS200 Curriculum (DevOps Platform Engineer — 399h): Production Kubernetes Gateway API implementations, advanced routing policies, high-performance container networking (SR-IOV / RoCE), real-time monitoring via Prometheus / Grafana, and declarative multi-cluster delivery with OpenShift GitOps (ArgoCD).

---

6. Sources & Technical References

• Red Hat Developer Blog — Intelligent inference scheduling with llm-d on Red Hat AI : developers.redhat.com/articles/2026/06/11/intelligent-inference-scheduling-llm-d-red-hat-ai
• Red Hat Developer Blog — Introduction to distributed inference with llm-d : developers.redhat.com/articles/2025/11/21/introduction-distributed-inference-llm-d
• Red Hat Blog — Demystifying llm-d and vLLM: The race to production : redhat.com/en/blog/demystifying-llm-d-and-vllm-race-production
• CNCF / llm-d Project Repository : github.com/llm-d/llm-d — documentation: llm-d.ai
• vLLM Documentation — Disaggregated Prefill & Decode Architecture : docs.vllm.ai/en/stable/features/disagg_prefill/
• Kubernetes Gateway API — Inference Extension Working Group : gateway-api.sigs.k8s.io