# llm-d & Distributed Inference on OpenShift: Prefill / Decode Disaggregation and Cache-Aware Routing > llm-d architecture on Red Hat OpenShift and vLLM: Prefill/Decode disaggregation, RDMA KV-cache transfer, and intelligent Gateway API routing for enterprise LLM workloads. - **Date de publication** : 2026-10-06T08:30:00.000Z --- Scaling enterprise generative AI inference encounters a fundamental architectural bottleneck: the mismatch between the compute profiles of the **Prefill** stage (prompt ingestion and Key-Value cache computation) and the **Decode** stage (sequential token generation). In standard monolithic vLLM deployments, both phases share the same GPU, leading to severe resource contention on long-context queries and unpredictable Inter-Token Latency (ITL) jitter. The **llm-d** architecture—a CNCF Sandbox project backed by **Red Hat**, **IBM Research**, **Google Cloud**, **CoreWeave**, and **NVIDIA**—standardizes Kubernetes-native orchestration for disaggregated inference (*Prefill/Decode Disaggregation*) and *KV-cache-aware routing* on **Red Hat OpenShift AI**. --- ### 1. The Monolithic Bottleneck: Conflicting Compute Phases LLM inference operates under two fundamentally different execution profiles: 1. **Prefill Phase (Compute-Bound):** Highly parallel processing of the input sequence to compute initial attention matrices and generate the KV cache. This phase saturates Tensor Cores and requires raw compute throughput (TFLOPS). 2. **Decode Phase (Memory-Bandwidth-Bound):** Autoregressive token-by-token generation. Each new token requires minimal matrix multiplication but heavily taxes High Bandwidth Memory (HBM) bandwidth to reload model weights and KV-cache tensors at every step. In a monolithic setup, large input prompts (e.g., 64k or 128k tokens in enterprise RAG pipelines) monopolize GPU cores, stalling token generation for concurrent active streams. As a result, **Time-To-First-Token (TTFT)** spikes and **Inter-Token Latency (ITL)** degrades. --- ### 2. Prefill / Decode (P/D) Disaggregation with vLLM Disaggregated inference splits the serving pipeline into two dedicated, independently scalable worker pools: * **Prefill Pool (Compute-Optimized Instances):** Ingests incoming prompts, computes the KV cache at maximum throughput, and immediately offloads the tensor state without running the token decode loop. * **Decode Pool (Memory-Bandwidth-Optimized Instances):** Ingests the pre-computed KV-cache tensor and streams output tokens sequentially with minimal, predictable latency. Inter-node KV-cache transfer is handled via high-speed interfaces (**vLLM NIXL / UCX / RDMA over Converged Ethernet**), completely eliminating redundant prompt reprocessing on decode nodes. --- ### 3. The Role of llm-d: Intelligent Scheduling and Gateway API Inference Extension **llm-d** delivers the Kubernetes-native control plane required to orchestrate disaggregated clusters: * **`InferencePool` and `InferenceObjective` CRDs:** Declarative specification of specialized compute pools (NVIDIA H100/A100, AMD Instinct, L40S) and explicit SLO targets (target TTFT, p99 token latency). * **Prefix-Cache-Aware Routing:** Operating via the **Kubernetes Gateway API Inference Extension**, the inference gateway maintains a distributed prefix index. Prompts sharing common prefixes (system instructions, recurring RAG documents) are automatically dispatched to pods that already hold cached KV blocks, cutting TTFT by up to 90%. * **Smart Load Balancing:** Dynamically evaluates GPU memory pressure and real-time active batch sizes across vLLM pods rather than relying on naive round-robin scheduling. --- ### 4. Operationalization on Red Hat OpenShift AI Deploying llm-d on **Red Hat OpenShift** leverages core enterprise platform primitives: * **KServe & ServingRuntime:** Declarative management and lifecycle operations for containerized vLLM serving workloads. * **Independent Autoscaling with KEDA:** Decoupled scaling of Prefill pods based on queue depth and token volume without artificially over-provisioning Decode capacity. * **High-Speed Networking (SR-IOV / RoCE):** Deterministic high-bandwidth networking enabling sub-millisecond KV-cache handoffs across cluster nodes. * **OpenTelemetry & Prometheus Observability:** Full visibility into golden inference metrics (`vllm:time_to_first_token_seconds`, `vllm:time_per_output_token_seconds`, cache hit ratios). --- ### 5. Curriculum Alignment & Skills at Aperta Scientia * **AS300 Curriculum (AI Platform Engineer — 399h):** Core alignment with distributed AI inference architectures—advanced **vLLM** runtime configuration, **KServe** ServingRuntimes under **Red Hat OpenShift AI (RHOAI)**, distributed KV-cache management, **llm-d** `InferencePool` CRDs, and inference performance benchmarking (GuideLLM / lm_eval). * **AS200 Curriculum (DevOps Platform Engineer — 399h):** Production **Kubernetes Gateway API** implementations, advanced routing policies, high-performance container networking (SR-IOV / RoCE), real-time monitoring via **Prometheus / Grafana**, and declarative multi-cluster delivery with **OpenShift GitOps (ArgoCD)**. --- ### 6. Sources & Technical References * **Red Hat Developer Blog — *Intelligent inference scheduling with llm-d on Red Hat AI* :** [developers.redhat.com/articles/2026/06/11/intelligent-inference-scheduling-llm-d-red-hat-ai](https://developers.redhat.com/articles/2026/06/11/intelligent-inference-scheduling-llm-d-red-hat-ai) * **Red Hat Developer Blog — *Introduction to distributed inference with llm-d* :** [developers.redhat.com/articles/2025/11/21/introduction-distributed-inference-llm-d](https://developers.redhat.com/articles/2025/11/21/introduction-distributed-inference-llm-d) * **Red Hat Blog — *Demystifying llm-d and vLLM: The race to production* :** [redhat.com/en/blog/demystifying-llm-d-and-vllm-race-production](https://www.redhat.com/en/blog/demystifying-llm-d-and-vllm-race-production) * **CNCF / llm-d Project Repository :** [github.com/llm-d/llm-d](https://github.com/llm-d/llm-d) — documentation: [llm-d.ai](https://llm-d.ai) * **vLLM Documentation — *Disaggregated Prefill & Decode Architecture* :** [docs.vllm.ai/en/stable/features/disagg_prefill/](https://docs.vllm.ai/en/stable/features/disagg_prefill/) * **Kubernetes Gateway API — *Inference Extension Working Group* :** [gateway-api.sigs.k8s.io](https://gateway-api.sigs.k8s.io) --- *Source : https://aperta.io/radar/2026-10-06-llm-d-distributed-inference-prefill-decode-disaggregation-vllm-openshift/ — Aperta Scientia Tech Radar*