# Event-Driven Ansible (EDA): Closed-Loop Auto-Remediation on Kubernetes

> How pairing Prometheus Alertmanager with Ansible Rulebooks automates node cordoning, pod draining, and worker reprovisioning without human intervention.

- **Date de publication** : 2026-09-09T08:30:00.000Z

---

In enterprise-scale Kubernetes and OpenShift environments, Mean Time to Remediation (**MTTR**) often hinges on human on-call reactivity: receiving alerts, performing manual triage, running maintenance scripts, and logging tickets.

Adopting **Event-Driven Ansible (EDA)** within Ansible Automation Platform (AAP) shifts infrastructure operations toward **closed-loop auto-remediation**, translating telemetry events directly into instant, policy-driven engineering workflows.

---

### 1. The Human Bottleneck in Platform Incidents
When a Kubernetes worker node experiences critical memory pressure (`NodeMemoryPressure`) or a degraded kubelet service:
* **Operational Latency:** Minutes pass between Prometheus alert firing and operator action, causing cascading application degradation (OOMKilled pods, traffic dropouts).
* **Execution Errors Under Pressure:** Manual `cordon` and `drain` commands executed under stress risk disrupting sensitive statefulsets and distributed caches.
* **Low-Value Toil:** Routine system restarts and disk space cleanups consume valuable engineering bandwidth that should be focused on platform features.

---

### 2. The EDA Architecture: Sources, Rules, and Actions
Event-Driven Ansible uses a declarative YAML specification called **Ansible Rulebooks**, built around three core primitives:
1. **Event Sources:** Direct integrations with Prometheus Alertmanager webhooks, Kafka streaming topics, or Vector log streams.
2. **Conditional Rule Engine:** Real-time event matching (e.g., `event.alert.alertname == "KubeNodeMemoryPressure" and event.alert.severity == "critical"`).
3. **Automated Action Handlers:** Triggering AAP Job Templates that programmatically:
   - Cordon the degraded node via the `kubernetes.core` collection.
   - Gracefully drain scheduled pods while strictly honoring `PodDisruptionBudgets`.
   - Execute target host playbooks (restarting container runtimes, purging system caches) or trigger automated node reprovisioning via GitOps/IPI.

---

### 3. Production Skills in Aperta Scientia AS200
At Aperta Scientia, event-driven infrastructure automation is taught hands-on:
* **AS200 Track (DevOps Platform Engineer — DO374):** Writing production-grade Ansible Rulebooks, securing webhook event ingestion, and integrating closed-loop workflows with OpenShift Monitoring.
* **Measurable Outcomes:** Slashing MTTR from 30+ minutes to sub-second automated responses, preserving strict enterprise SLO commitments.

---

### 4. Technical Sources & References
* **Event-Driven Ansible Official Documentation:** [Red Hat Ansible Automation Platform Guide](https://docs.redhat.com/en/documentation/red_hat_ansible_automation_platform/)
* **Ansible Rulebook Upstream Project:** [ansible-rulebook Documentation](https://ansible.readthedocs.io/projects/rulebook/)
* **Event-Driven Automation with Prometheus & OpenShift:** [Red Hat Developer Blog](https://developers.redhat.com/articles/2023/06/20/event-driven-ansible-closed-loop-automation)

---

*Source : https://aperta.io/radar/2026-09-09-event-driven-ansible-closed-loop-remediation-kubernetes/ — Aperta Scientia Tech Radar*