Splunk

How Splunk Unified Security Observability and Reduced Infrastructure Costs with Cillium’s Tetragon

Splunk is a data analytics and observability platform that helps organizations search, monitor, and analyze machine-generated data from IT systems, security infrastructure, and business applications. It is now part of Cisco, which acquired Splunk in 2024 to strengthen its security and observability portfolio. 

Challenge

Fragmented Tools Left Security Gaps at Cloud Scale 

Splunk operates approximately 170,000 cloud resources across AWS, Azure, and Google Cloud to support its software development pipelines, Splunk Cloud, Splunk Observability Cloud, and Splunk AppDynamics, which run across both virtual machines and Kubernetes deployments. Securing an environment of this scale required running multiple security services in parallel, and the team began to struggle with consistent security coverage due to tool fragmentation. 

The team had been running three separate tools to cover process auditing and security observability. None of these were Kubernetes-aware, which meant each had to be configured, maintained, and supported independently. Security policies also often required entirely different implementations inside virtual machines versus Kubernetes clusters. The result was a growing operational overhead where the Security Engineering team carried a heavy maintenance burden while coverage remained inconsistent across the environment. 

There were also struggles at the infrastructure level. In order to achieve observability in Kubernetes, the team had to add multiple daemonsets, with each pod consisting of multiple containers, plus an OpenTelemetry collector sidecar, to send the data to Splunk. Across the Kubernetes clusters running Splunk Cloud Services, that per-pod overhead accumulated quickly. 

Splunk’s security team realized it needed a solution that could span both worlds. It needed to be cloud native and Kubernetes-aware enough to eliminate sidecar overhead, but flexible enough to extend across its VM environments, all while providing clean, actionable data for process execution, filesystem activities, and network connections. 

“We were running multiple security services that each had to be maintained separately, and none of them had any awareness of Kubernetes. Every time we needed to change a policy, we were doing it twice, once for VMs and once for Kubernetes clusters. It wasn’t sustainable at our scale.” -– Richard Wilhite, Senior Staff Security Engineer, Splunk 

Solution

Tetragon Replaces Legacy Overhead with Visibility 

Splunk deployed open source Tetragon (a sub-project of Cilium) across the Kubernetes clusters running Splunk Cloud Services to serve as the foundation of its runtime security observability program, replacing three existing tools. Tetragon’s eBPF-based architecture gave Splunk deep visibility into process execution, file activity, and network connections across its Kubernetes environment. It also runs at the node level, removing the convoluted legacy daemonsets that had previously bogged down the security observability workload. 

“Replacing our existing daemonset and sidecar-based telemetry with Tetragon gave us better security data and dramatically reduced the infrastructure overhead across Splunk Cloud Services.” – Richard Wilhite, Senior Staff Security Engineer, Splunk 

On Kubernetes, Helm charts integrated into CI/CD automation handle Tetragon deployment and tracing policy management across clusters. Splunk is now extending the same approach to its virtual machine environments: Puppet will manage Tetragon deployment across 80,000 systemd VMs running Splunk Enterprise, with 31,000 Splunk AppDynamics VMs and 60,000 Splunk Observability resources to follow in a later phase. 

Tetragon automatically enriches runtime events with Kubernetes API metadata, including pod names, namespaces, and labels, without requiring manual correlation. This context-rich telemetry is forwarded to Splunk’s security analytics platform, where the Detection Engineering team uses it to build detections for Splunk’s own software, generate alerts for the Security Operations Center (SOC), respond to emerging threats, and enrich events with CVE data by combining Tetragon output with other security data sources. 

Impact

Security Coverage Improves as Infrastructure Costs Drop 

Tetragon gave Splunk’s security team consistent, Kubernetes-aware observability over the production infrastructure running Splunk Cloud Services, something the previous collection of tools could not uniformly provide. 

Eliminating the daemonset and sidecar model delivered immediate and measurable infrastructure cost improvements. CPU utilization dropped by 67% and memory utilization dropped by 74%, with no reduction in security coverage. The team gained better data while running a lighter infrastructure footprint. 

The quality of the underlying security data also improved, and the Detection Engineering team has been able to enrich events with CVE data in ways that were not previously possible. The ability to correlate Kubernetes API metadata with process, file, and network events has shortened root-cause analysis workflows, informed response actions, and given analysts clearer boundaries around incident scope. 

The consolidation of three fragmented tools into a single solution reduced the operational maintenance burden across both Kubernetes and VM environments. With Splunk Cloud Services’ Kubernetes environment secured, the Security Engineering team is now extending the same Tetragon deployment and policy framework to Splunk Enterprise’s VM infrastructure, with Splunk AppDynamics workloads and Splunk Observability to follow across all three cloud providers. 

“Tetragon gave us a single, consistent security layer across our entire Kubernetes environment for the first time. The fact that we can now extend the exact same policies to our VM fleet means we are finally working toward a unified security posture across all of Splunk’s cloud infrastructure.” -– Richard Wilhite, Senior Staff Security Engineer, Splunk 

*Tetragon is a sub-project of CNCF graduated project, Cilium.