Splunk and Red Hat collaborate to automate response to observability alerts for improved AIOps

Event-Driven Ansible, part of Red Hat Ansible Automation Platform, automates actions to enable AIOps scenarios and deliver greater speed, consistency and resilience when responding to issues and alerts. Splunk, part of Cisco Systems, offers a widely-adopted observability portfolio designed to help organizations understand their digital systems, detect threats and improve operational efficiency. 

In collaboration with Red Hat and Cisco, Splunk can accelerate and simplify the creation of automated response scenarios for Splunk alerts. Joint customers can now more easily automate full responses, from alert to action, which provides benefits like fewer service tickets, faster mean time to resolution (MTTR) and better resilience, including rapid response to security alerts or remediation of issues impacting the availability of key applications. 

Benefits of the integration

Like all capabilities in Ansible Automation Platform, Event-Driven Ansible is highly flexible. Users can select specific alerts and design the desired response – from automatically creating tickets and notifications, to read-only fact gathering, codified troubleshooting, completing automated IT management steps and calling trusted Ansible Playbooks. Customers can use the Splunk IT Service Intelligence (ITSI) observability solution’s Event Analytics capabilities with Event-Driven Ansible to more quickly correlate and identify business-critical issues and automate responses in an AIOps model. 

As an AIOps, analytics and IT management solution, Splunk ITSI helps teams prevent incidents before they impact customers. Using AI and rules, Splunk ITSI correlates data collected from monitoring sources and delivers a single live view of relevant IT and business services, reducing alert noise and proactively preventing outages.

"We see a lot of customer value across Splunk ITSI and Ansible Automation Platform users, so this collaboration is valuable to our joint customers. By combining the power of Splunk with Event-Driven Ansible, we’re helping customers take faster, smarter actions through automation to keep their systems resilient and their teams more agile," said Anush Jayaraman, Director of Partner Solutions Engineering, Splunk, a Cisco Company.  

Splunk has also harnessed data across the Cisco ecosystem, turning data from Cisco ThousandEyes, Catalyst Center and Meraki to help organizations better connect the dots across their digital stack—from application to infrastructure to network. Using Event-Driven Ansible, organizations can understand the “bigger picture” across their stack–and then respond to changing conditions and issues quickly.

How event-driven automation works with Splunk

So what has been developed to make automated response scenarios faster and easier to deliver? A Splunk add-on is now available on Splunkbase, allowing alerts to be sent to Event-Driven Ansible using either webhooks or Kafka. Red Hat is supporting this add-on for customers with a Red Hat Ansible Automation Platform subscription. This article, published on Splunk’s community-focused Lantern web site, explains how to get started.   

Let’s dig into how it works. The figure below describes the receive-decide-respond model in Event-Driven Ansible. In the decide phase, Ansible Rulebooks play a key role. These rulebooks are written using similar methods to Ansible Playbooks, but they include conditional rules. When an alert is received, the event is evaluated by the conditions in the rulebook. When conditions are met, the desired action is triggered for response or resolution of the alert. Actions may include calling Ansible Playbooks, modules or executing automation templates in Ansible Automation Platform.

So how does this scenario work with a Splunk alert? Figure 2 is a more detailed view of the receive-decide-respond scenario shown above. You will see the add-on in use as well as Splunk ITSI and Splunk Enterprise Security where alerts are being generated with or without an AI model.    

Expanding use of event-driven automation with Splunk

Both Red Hat and Splunk envision a growing maturity model for adoption. We support a “start small, think big” approach. Your first event-driven automation tasks should be simple, then grow in scope and sophistication to provide benefits like better work-life balance and more time for innovation. Your first event-driven automation tasks might be automatically creating tickets and notifications, or automatically renewing certificates, especially when they expire during the overnight hours. 

Once you see these benefits, you can grow from there. For example, you may build rulebooks to shut down or redirect traffic around an area where there is a security threat. Or, you can build and trigger threshold responses such as in an e-commerce high-traffic scenario. Here are some additional ideas of how to increase scope and sophistication of alert-response scenarios:

• Automated incident response
• Compliance and configuration drift remediation
• AI/ML-driven remediation loops
• Infrastructure provisioning visibility
• Security Operations Center (SOC) workflows
• Change management auditing
• Closed-loop automation

Conclusion and resources

Red Hat and Splunk are excited about this collaboration and what is to come. We invite you to give it a try using the resources below. Please share your thoughts with either Red Hat or Splunk so we understand what you need and how you are benefiting. Watch the Event-Driven Ansible web page for news and resources around this collaboration.  

Additional resources:

• Splunkbase add-on
• Splunk Lantern how to technical article
• Event-Driven Ansible web page
• Event-Driven Ansible interactive lab
• Video demo: art of the possible for AIOps
• Cisco and Splunk Strengthen Enterprise Digital Resilience in the AI Era