# Docling, Granite & OpenShift AI: Multimodal Ingestion, Guided Decoding, and Structured RAG Extraction > Document ingestion with Docling layout vision models, vLLM guided decoding vs defensive guardrails with IBM Granite on Red Hat OpenShift AI (RHOAI). - **Date de publication** : 2026-10-13T08:30:00.000Z --- In enterprise Retrieval-Augmented Generation (RAG) and sovereign AI architectures, the fidelity of downstream vector search directly hinges on document ingestion accuracy. Enterprise knowledge remains heavily locked inside static office formats (PDFs, Word documents, spreadsheets). Naive parsing pipelines consistently fail on a fundamental systems mismatch: the stark difference between **printing order** and **semantic reading order**. In a detailed technical report published on the **Red Hat Developer Blog**, platform engineering teams outline the architecture of an AI-driven matching engine combining the open source **Docling** library (developed by IBM Research and Red Hat), open-weight **IBM Granite** foundation models, and **Red Hat OpenShift AI (RHOAI)**. This design provides essential engineering insights into the trade-offs between **grammar-based guided decoding** and **defensive guardrails** when extracting structured data from unstructured enterprise documents. --- ### 1. The Document Representation Gap: Printing Order vs Reading Order The PDF specification was fundamentally designed for graphical device printing fidelity rather than semantic data extraction. A raw PDF content stream is orchestrated by graphical matrix operators (such as the `Tm` text matrix operator that positions the cursor at precise Cartesian coordinates `X` and `Y`). Consequently: * The sequence in the raw binary content stream reflects the print layout execution order rather than human reading hierarchy. * A headline or section header can be serialized after a sidebar block, page footer, or metadata chunk. * Multi-column text layouts, complex tables, and captioned figures disintegrate into disconnected text spans when parsed using basic text extractors or naive OCR tools (`pypdf`, regex tokenizers). This structural degradation directly pollutes chunking strategies in RAG pipelines, creating incoherent chunks that wash out cosine similarity metrics and inject corrupted context into downstream LLMs. --- ### 2. Docling's Hybrid Ingestion Architecture: Layout Vision & Intermediate Models To bridge this semantic divide, **Docling** implements a hybrid architecture combining specialized document parsers (covering 35 formats via `python-docx`, `openpyxl`, and `BeautifulSoup`) with deep learning vision models for Document Layout Analysis (DLA). For PDF processing: 1. **Vision-based layout segmentation:** Docling leverages a layout neural network trained on over 150,000 annotated documents (such as `docling-layout-heron`) to predict precise bounding boxes and assign semantic element labels (`[section_header]`, `[text]`, `[table]`, `[caption]`, `[furniture]`). 2. **Reverse coordinate projection:** The raw `Tm` coordinate stream of the PDF is mapped back into the predicted layout bounding boxes, accurately reconstructing the original human reading order. 3. **`DoclingDocument` intermediate representation:** The recovered hierarchy is mapped into an intermediate Pydantic-based object model. This model retains ordered `TextItem` collections, table matrices, and hierarchical tree relationships (body, sections, nested sub-sections). This structured intermediate state is subsequently exported into **clean, sanitized Markdown**, giving foundation models pristine input devoid of formatting noise. --- ### 3. Structured Data Extraction: Guided Decoding vs Defensive Guardrails Once documents are converted into clean Markdown, the next architectural stage extracts strictly typed JSON schemas using compact, high-efficiency models like **IBM Granite 3.1** or **Granite 4:3B**. Two architectural strategies govern schema compliance: #### Strategy A: Runtime Guided Decoding Supported natively in inference runtimes like **vLLM** via specialized libraries (`llguidance`, `xgrammar`), guided decoding applies a logit mask dynamically during token sampling. Tokens that violate the target JSON Schema grammar are eliminated from the candidate probability distribution. * **Advantage:** Guarantees 100% structural and syntactic compliance on output with zero post-processing. * **Production vulnerability (Forced Hallucination):** When a required schema property (such as `years_of_experience`) is entirely missing from the input document, strict grammatical masking prohibits the model from omitting the key or returning a null value. Forced to comply with the schema constraint, the model **hallucinates a plausible value**, producing clean but factually invalid data. #### Strategy B: Defensive Guardrails with Remediation Loops The approach implemented in resilient architectures (such as Red Hat's PeopleMesh reference architecture) allows natural model emission followed by deterministic post-generation validation: * Stripping markdown code fence wrappers (```json). * List deduplication, string trimming, and canonical whitespace normalization. * Strict clamping against domain-specific controlled vocabularies (e.g. mapping unknown seniority designations to default fallback values). * Pydantic schema validation: if critical attributes are absent from source text, the schema explicitly captures `null` states rather than fabricating facts. When schema parsing fails, a deterministic retry loop passes structured validation errors back to the model for correction. --- ### 4. Focused Vectorization and Preventing Embedding Dilution A common design flaw in RAG pipelines is projecting an entire document or multi-attribute profile into a single dense vector embedding. This creates severe **Embedding Dilution**: mathematically averaging dozens of distinct technical skills, domain experiences, and project scopes into a single 1536-dimensional vector washes out high-value discriminatory signals in latent space. The resultant embedding migrates toward an undifferentiated centroid. To ensure pinpoint retrieval accuracy: * The system decomposes entities into modular, domain-specific semantic facets (primary role, core technical competencies, business domain). * Each facet is embedded independently or indexed using hybrid search (dense vector retrieval paired with scalar metadata filtering and sparse BM25 lexical search) running on **PostgreSQL with pgvector** or dedicated distributed engines like **Milvus**. --- ### 5. Production Operations on Red Hat OpenShift AI (RHOAI) Operationalizing this pipeline on enterprise infrastructure requires core **Red Hat OpenShift** platform primitives: * **NVIDIA GPU Operator:** Hardware-accelerated runtime scheduling for Docling layout vision models, enabling multi-page document parsing at line-rate throughput. * **vLLM & KServe ServingRuntimes:** Declarative, cloud-native lifecycle management for IBM Granite models on OpenShift AI, featuring optimized KV-cache management, dynamic adapter loading, and high-concurrency serving endpoints. * **Pipeline Automation via Data Science Pipelines (Kubeflow Pipelines v2):** End-to-end declarative orchestration of document ingestion, schema validation, and vector database indexing. * **Sovereign Persistence with OpenShift Data Foundation (ODF):** Scalable S3 object storage via NooBaa for raw document archives, paired with encrypted Ceph RBD persistent volumes for low-latency vector database state. --- ### 6. Curriculum Alignment: Aperta Scientia * **AS300 (AI Platform Engineer — 399h):** This production pattern directly maps to enterprise AI data platform engineering — multimodal document ingestion with **Docling**, deployment and configuration of **vLLM** ServingRuntimes with **IBM Granite** on **Red Hat OpenShift AI (RHOAI)**, operational evaluation of *Guided Decoding* (`llguidance`) versus deterministic guardrails, preventing embedding dilution in enterprise RAG systems, and vector database integration with **Milvus / ODF**. * **AS200 (DevOps Platform Engineer — 399h):** GitOps delivery of AI ServingRuntimes using **OpenShift GitOps (ArgoCD)**, lifecycle governance of the **GPU Operator**, multi-tenant namespace isolation for compute-intensive ingestion workloads, and zero-trust API ingress security. --- ### 7. Sources & Technical References * **Red Hat Developer Blog — *Build an AI matching engine with Docling, Granite & OpenShift AI*:* [developers.redhat.com/articles/2026/10/12/build-ai-matching-engine-docling-granite-openshift-ai](https://developers.redhat.com/articles/2026/10/12/build-ai-matching-engine-docling-granite-openshift-ai) (accessed October 13, 2026) * **Docling Layout Analysis Research Paper (arXiv:2509.11720):** [arxiv.org/abs/2509.11720](https://arxiv.org/abs/2509.11720) * **Docling Project & Model Repository:** [github.com/DS4SD/docling](https://github.com/DS4SD/docling) — models: [huggingface.co/docling-project/docling-layout-heron](https://huggingface.co/docling-project/docling-layout-heron) * **vLLM Documentation — *Structured Outputs (llguidance & xgrammar)*:** [docs.vllm.ai/en/stable/features/structured_outputs/](https://docs.vllm.ai/en/stable/features/structured_outputs/) * **Red Hat OpenShift AI (RHOAI) Overview:** [redhat.com/en/technologies/cloud-computing/openshift/openshift-ai](https://www.redhat.com/en/technologies/cloud-computing/openshift/openshift-ai) --- *Source : https://aperta.io/radar/2026-10-13-docling-granite-rhoai-structured-rag-extraction/ — Aperta Scientia Tech Radar*