Aperta Scientia Crest
← Back to Tech Radar
📡 Aperta Intelligence Lab
Event-Driven Ansible: Closed-Loop Remediation and Self-Healing on OpenShift

Event-Driven Ansible: Closed-Loop Remediation and Self-Healing on OpenShift

📅 October 1, 2026
Tech Radar Cloud & DevOps Automation

Briefing Summary

An Ansible playbook launched by hand at 2 a.m. during an incident is not automation: it is a human script with human latency. The real change of nature is turning a signal — a Prometheus alert, a Kubernetes event, a Kafka message, a journald log line — into a decision and an action, without waking anyone up. That is exactly the scope of Event-Driven Ansible (EDA): the source → rule → action triptych, materialised in a versioned rulebook.

With the arrival of Ansible Automation Platform 2.7 and its MCP server, EDA moves out of the demo-gadget register and becomes a platform building block: the control plane that connects observability to remediation, and now a conversational agent on that same control plane.

1. The source / rule / action model

A rulebook is a YAML artifact, versionable and reviewed in a pull request, deliberately minimal in structure:

---
- name: Listen for events on a webhook
  hosts: localhost
  sources:
    - ansible.eda.webhook:
        host: 0.0.0.0
        port: 5000
  rules:
    - name: Say Hello
      condition: event.payload.message == "Ansible is super cool"
      action:
        run_playbook:
          name: say-what.yml

Three blocks, three responsibilities:

  • sources: an event source provided by the ansible.eda collection — webhook, alertmanager, kafka, file, journald, pg_listen, range for testing, plus vendor integrations (Splunk, ServiceNow, and more). Every source exposes a stream of normalised events.
  • rules / condition: filtering, expressed in EDA’s condition DSL over the event object (attribute access, comparison and logical operators, membership tests). Business logic lives here — and so does the maintenance debt.
  • action: run_playbook, run_module, run_job_template (triggering an AAP Controller Job Template, thus reusing existing RBAC and inventory), or debug/print_event for observation.

An important architectural point: the condition is evaluated against the event, not against system state. A badly written rulebook therefore replays the same remediation in a loop on a flapping alert. Mastering the blast radius is not optional.

2. From the CLI to the control plane: EDA Controller and Decision Environments

ansible-rulebook is a local executable, useful for prototyping and debugging. In production, EDA is operated by the EDA Controller (ansible/eda-server, Apache-2.0) integrated into Ansible Automation Platform: rulebooks become activations (enable / disable / restart), backed by a Git project and an inventory.

The execution substrate is the Decision Environment: a container image (EE) embedding ansible-rulebook, Python and the required collections. It is the decision-side counterpart of the Execution Environment on the execution side.

  • Reproducibility: the same built, versioned DE pushed to a registry (Quay, Automation Hub) guarantees that the decision taken in staging is identical to the one taken in production.
  • Isolation: one DE per domain (network, security, platform) limits the collections available, and therefore the action surface of a compromised rule.
  • Traceability: every activation produces an executable history — which event, which rule, which action, which result. That is the raw material for audit and post-mortem.

A rulebook is not tested in production. The ansible.eda collection ships synthetic sources (range) that let you replay an event stream in CI before any activation.

3. Closed-loop remediation: Alertmanager as the trigger

The canonical use case on OpenShift is the Prometheus → Alertmanager → EDA → cluster action loop: the PrometheusRule fires, Alertmanager routes and notifies through a webhook receiver, the ansible.eda.alertmanager source ingests the payload, the rule filters on alert name and labels, the action runs the remediation playbook.

A reference implementation documented by the Red Hat Developer Blog deploys an instrumented Quarkus application, a ServiceMonitor, Alertmanager with its AlertmanagerConfig, Prometheus, and an EDA Pod carrying the rulebook and the action playbook — entirely declaratively, with oc apply.

What separates a demo from an operable system:

  • Idempotence: remediation must be a no-op if the state is already correct. A Pod restart loop is an incident, not a cure.
  • Window and debounce: never react to a firing alert on a transient symptom; require a minimum duration or a for: clause in the Prometheus rule.
  • Graduated escalation: self-correction for known, safe cases, ticket creation or notification otherwise. EDA is not “automate everything”, it is automating what is decidable.
  • Least-privilege RBAC: the ServiceAccount carrying the remediation should only be able to do what it must (draining a node is not deleting a PVC).

4. Platform use cases that survive production

  • Node lifecycle: cordon and drain a node flagged as degraded by its metrics, before the kubelet does it for you, with eviction honouring application PDBs.
  • Certificates and secrets: detecting a certificate nearing expiry, rotating it and reloading the affected components — a classic that should never depend on a human.
  • Elasticity: business-signal-driven autoscaling, including on virtualised workloads (the Red Hat Developer Blog documents EDA-driven automatic CPU scaling of OpenShift Virtualization machines).
  • Security (SOAR): isolating a suspicious workload on detection, quarantining, enriching and opening a ticket. Closing the gap between detection and containment remains the main lever for reducing exposure time.
  • AIOps: chaining observability and ITSM signals (Instana, Splunk, ServiceNow) into instrumented remediation — the subject of the solution guides published for AAP 2.7.

5. AAP 2.7: EDA in the agentic era

Ansible Automation Platform 2.7 changes EDA’s ergonomics and integration more than its core:

  • MCP server for AAP (Technology Preview): external AI agents (Claude, Cursor, and others) query job status, gather facts and launch automation workflows in natural language. The control plane becomes a capability callable by an agent — and therefore a security object in its own right.
  • Automation intelligent assistant with bring-your-own-knowledge: the assistant’s RAG pipeline can be extended with the organisation’s internal documentation (policies, runbooks, terminology), bringing the assistant closer to real operational context.
  • OIDC identity provider for HashiCorp Vault (Technology Preview): AAP issues short-lived JWTs, scoped to the job, the user and the organisation, replacing long-lived service accounts. This is exactly the model required for a rulebook to access a secret without a stored secret.
  • Execution Environment builder & content catalog: the decision environment is built in the UI from automatically discovered collections — EDA’s barrier to entry drops, and the risk of drift rises accordingly.
  • Ansible development workspaces: a standardised browser-based development environment (VS Code, ansible-core, ansible-creator, ansible-lint, Molecule).

The engineer’s read: giving an LLM agent the ability to launch remediation only makes sense if the guardrails — RBAC, rulebook review, immutable DE, idempotence — are already in place. The agent amplifies the quality of the foundation, it does not replace it.

6. What must be taught: EDA as an operations discipline

Event-Driven Ansible is not a product to add to a catalogue: it is the junction between observability, configuration management and operational security. The skill the market lacks is not “writing a rulebook”, it is designing a bounded decision loop: which event, which threshold, which action, which rollback, which journal.

7. Skills & Aperta Scientia Curriculum

  • AS200 track (DevOps Platform Engineer — 399h): this topic feeds directly into infrastructure automation and fleet-scale operations — writing EDA rulebooks and activations on the DO374 / AAP side, integration with Prometheus/Alertmanager observability and security policies, multi-cluster operations and hardening on the DO380 side, and mastery of Execution / Decision Environments as delivery-chain artifacts. Prepares for the EX188, EX280 and EX288 certifications.
  • AS300 track (AI Platform Engineer — 399h): transposing the model to the AI control plane — reactive remediation on KServe InferenceService and vLLM ServingRuntime objects (evicting a faulty GPU, shedding load on a degraded ServingRuntime, controlled restart of an MLOps pipeline), and interacting with the AAP MCP server as an integration point between agents and automation. Self-healing an inference platform is an operations skill, not an option.

8. Sources & Technical References

Apply these technologies in production

Explore our intensive 3-month AS200 (DevOps) and AS300 (AI Platform Engineer) curriculums.

View Curriculums →