Celonis

How Celonis Standardized Networking Across 150+ Kubernetes Clusters with Cilium

Challenge

Celonis is the global leader in process intelligence. Industry-leading companies around the world trust Celonis and its global ecosystem of partners to make their AI agents, people, systems, and processes work together to achieve transformational outcomes. Moving that volume of data at scale makes networking a foundational priority.

A platform engineering team of around 30 engineers runs Celonis’ infrastructure across AWS, Azure, and GCP with more than 150 Kubernetes clusters in 22 cloud regions worldwide. Each customer environment runs as a fleet of three clusters: a main cluster, a machine learning cluster, and a query engine cluster. The infrastructure processes roughly 3.5 terabytes of data and 360 million requests every day.

After six years of rapid growth, the networking infrastructure had become fragmented across six Kubernetes distributions and three clouds. The fleet spanned EKS running AWS VPC CNI, AKS running Azure CNI, GKE running its own network stack, Gardener running Weave, kops running Flannel, and OpenShift running its built-in SDN with OpenVSwitch. On top of the CNI fragmentation, each environment ran a different mTLS implementation. 

“The engineering tax of operating this fragmented networking stack was high,” said Marcelo Mello, Platform Engineer at Celonis. “Every network issue meant context-switching between different stacks, different debugging tools, different limitations.”

Additionally, parts of the stack were running software that had reached end of life. This meant upgrading was often impossible because APIs were frozen or incompatible with Kubernetes versions still in use across the fleet, with some clusters still running retired Kubernetes versions. Even routine tasks like upgrading cert-manager to a supported version were blocked.

“We got to a point where we couldn’t upgrade core platform components because of Kubernetes version constraints,” sail Mello. “The fragmentation wasn’t just a networking problem. It was slowing down the entire platform.” 

The team launched an internal initiative called Project Bedrock to modernize how Celonis runs Kubernetes. Finding a common networking standard across every cloud and environment was one of the project’s core objectives.

Solution

After evaluating the alternatives, the team selected Cilium as the unified CNI for the entire stack. The networking options from AWS, Azure, and GCP were the obvious comparison point, but adopting them would have meant maintaining three separate networking stacks and three separate sets of operational expertise, keeping the same fragmentation problem the team was trying to solve.

Cilium offered something the managed CNI options could not: a single operational model across all three clouds, with networking, security, and observability in one coherent stack. The specific capabilities that made the case were L7 and FQDN-based network policies, Cluster Mesh as a path to multi-cluster connectivity, network observability from Hubble, and native WireGuard encryption to support HIPAA compliance requirements. The project’s CNCF graduated status and broad industry adoption mattered too. The team needed to commit to a long-term standard, not an experiment.

“Cilium checked all the boxes we were looking for,” Mello said. “It gave us the observability stack through Hubble, the performance we needed from eBPF in the kernel, and it’s the market standard. We did not want to make a bet.”

The implementation strategy varied by environment. For Azure, the team had a more straightforward path. The Azure portion of the fleet was largely running on OpenShift, and since Project Bedrock required moving off OpenShift regardless, the team provisioned new AKS clusters with Cilium from the start. 

For EKS, the team needed a more involved approach. In 2024, EKS did not support deploying a cluster without a pre-installed CNI, so the team designed a two-stage strategy to migrate the existing fleet to full eBPF mode and kube-proxy replacement, in order to use Cluster Mesh and L7 policies, without any customer downtime. First, the team installed Cilium alongside AWS VPC CNI in hybrid mode, labeling nodes and DaemonSets so that rolling deployments landed on Cilium-managed nodes while legacy nodes drained over hours or days. Once all workloads had moved, the team flipped each cluster to exclusive mode, removing AWS VPC CNI and kube-proxy entirely and leaving Cilium as the sole data plane.

“The migration itself was the hardest part,” said Mello. “We could not stop production clusters, so we had to design a path that let us move a live fleet without our customers ever noticing. You simply do not find ready-to-use migration instructions in public documentation for your exact scenario.” 

Impact

By standardizing on Cilium across the fleet, the platform team now operates from a single networking, observability, and security model regardless of which cloud provider a cluster runs on. On-call engineers no longer need to context-switch between CNI toolsets when investigating network issues.

The team also replaced IP allow-list automation with FQDN-based and L7 network policies, giving development teams substantially more precise access controls. Where teams previously maintained pipelines to manage and mutate IP blocks, they can now express policies that match on domain name patterns. For example, the team can restrict a workload to only S3 buckets within a specific AWS region using an FQDN match rather than a list of IP ranges that requires ongoing maintenance.

The team also enabled WireGuard transparent encryption across all clusters, satisfying the encryption requirements for HIPAA environments. Rolling it out required working closely with application teams whose socket behavior changed under full eBPF mode, but the result is encryption that is on by default with no application changes required.

For incident response, the team adopted Hubble as standard tooling. Mello created an internal Hubble cheat sheet for the SRE team, and the tool now appears consistently in incident notes when engineers are diagnosing connectivity issues.

“Cilium made my job genuinely easier,” said Mello. “Hubble tells me what’s happening on the network in seconds. L7 policies let me write security rules that match how services actually talk to each other, not how IPs happen to be assigned that day. That’s the difference between guessing and knowing.”

Looking ahead, the team is expanding Cluster Mesh across the fleet and evaluating Cilium’s ztunnel integration for environments where full pod-to-pod mTLS is required. The team is also incorporating Oracle Cloud clusters that will follow the same Cilium strategy already in place on AWS and Azure.