Transforming Scientific Computing Infrastructure for IPv6 with Cilium at ESnet
Challenge
ESnet, the data circulatory system for the U.S. Department of Energy faced high operational overhead from running individual VMs for a single containerized application with no unified observability, security, or deployment platform, while navigating federal IPv6-only mandates and incredibly high data throughput demands.
Solution
The platform team at ESnet implemented Kubernetes for workload consolidation, leveraging Cilium for eBPF-based networking, identity-based security, and IPv6-native BGP route advertisement in its on-premises infrastructure.
Impact
ESnet consolidated its networking toolset, simplifying deployment, increasing observability with Hubble, and creating a blueprint for IPv6-only Kubernetes deployments across national laboratories.
Projects used
By the numbers
100s
of VM deployments consolidated into 2 multi-tenant clusters
IPV6-ONLY INFRASTRUCTURE
supporting federal mandate compliance
UNPRECEDENTED OBSERVABILITY
through Hubble integration
ESnet, the Energy Sciences Network, serves as the data circulatory system for the U.S. Department of Energy, operating a high-performance research network backbone out of the Lawrence Berkeley National Laboratory that interconnects 17 national laboratories and 28 user facilities, including four supercomputing facilities.
The organization’s mission is to enable and accelerate scientific discovery by delivering unparalleled network infrastructure. ESnet ensures that scientific progress remains unconstrained by the physical location of data and compute resources and that petabytes of data can be transferred within hours. ESnet enables scale that commercial ISPs typically don’t handle for scientists running particle accelerators, supercomputers, climate models, genomics research, and astrophysics projects.
Seeking a Unified Approach for IPv6-Only Networking
ESnet faced operational overhead from a fragmented infrastructure approach. The organization was deploying applications across hundreds of individual virtual machines, each running in a single containerized application.
“We have a lot of applications where we’ll spin up a VM and deploy a containerized workload, and do it over and over again. It added unnecessary overhead.” Luke Baker, Group Lead of ESnet’s Platform Engineering Team
This approach created multiple challenges. There was no unifying platform for observability or security across deployments. They used security tools embedded in the systems, relying on iptables, user accounts, and sudo to manage access controls. It was functional but inconsistent across applications. ESnet needed unified deployment schemes and consistent tooling for developers across all deployment environments.
Another unique challenge for ESnet was the federal mandate OMB M-21-07, requiring all federal entities to be IPv6-capable by 2025. As Kapil Agrawal, Platform Security Engineer for ESnet noted: “At ESnet, networking is our bread and butter and we have had a long track record of leading the charge on IPv6. We have a data center that’s built IPv6-only.” When ESnet requested compute infrastructure for their new clusters, they received IPv6-only resources, making their networking requirements even more specialized.
Consolidating and Automating Advanced Networking with Cilium
ESnet implemented Kubernetes with Cilium to consolidate workloads and modernize their infrastructure. The team deployed two production clusters with specific workload profiles, enabling true multi-tenancy while avoiding the cluster-per-application anti-pattern they’d experienced with just running containers in VMs.
Cilium’s eBPF-based approach addressed ESnet’s unique networking requirements. “We were looking for a tool that would make cluster networking simple, but also has advanced capabilities available,” said Baker. “Cilium provides an on-ramp for getting started and securing clusters, but also offers deep networking related features like BGP and Cluster Mesh.”
The implementation leverages Cilium’s BGP control plane for automatic route advertisement. When worker nodes join clusters, they automatically establish BGP peering with leaf switches in ESnet’s data center. Cilium’s load balancer IPAM auto-assigns IPv6 /64 prefixes for each application, and BGP automatically announces these prefixes to the data center infrastructure.
Cilium allowed ESnet to unify multiple networking components into a single tool. “All those individual one-off things that you would need to build a cluster are now available as one component that you install,” noted Agrawal. This included CNI, load balancer, ingress, Gateway API, plus observability through Hubble and security tools like Cilium Host Firewall.
Moving from IP to Identity-Based Security
“From a security point of view, workload isolation was one big thing we were looking for,” explained Agrawal. “There was also a reckoning early on that IP addresses and hostnames in Kubernetes are ephemeral in nature, and our current security tooling treats IP addresses and hostnames as assets. That model doesn’t work with cloud native environments.”
Cilium solved this with its identity-based security approach. Instead of relying on IP addresses or hostnames, Cilium assigns security identities to each workload using SPIFFE/SPIRE, enabling fine-grained policies that bring security controls closer to the applications allowing for centralized visibility.
Achieving Unprecedented Control, Security, and Observability
ESnet consolidated its VM deployments into two production-grade clusters and is routing new applications into these clusters. This creates what Agrawal described as “an unprecedented level of control, security, and observability.” The networking improvements delivered significant operational benefits and Hubble, Cilium’s observability platform, became a critical tool for network debugging.
“The power of Hubble is a godsend. If there’s one tool I use day in and day out, that’s Hubble.” – Kapil Agrawal, Platform Security Engineer, ESnet
Creating a Blueprint for IPv6-Only Networking
ESnet’s success demonstrates that IPv6-only Kubernetes deployments are not only viable, but also work exceptionally well at government scale. This has provided a blueprint for other national laboratories facing similar mandates. The organization is sharing their learnings across the national lab complex, helping peers implement similar solutions.
Baker emphasized the broader impact: “We came into this need for a better way to deploy container workloads. The outcome was that we’ve identified there are usable patterns and real software that can support IPv6-only usage. We are now engaging with our peers across the national lab complex to describe how you can use Cilium and what IPv6-only looks like.”
Both Baker and Agrawal also recognized the Cilium community’s willingness to take feedback and incorporate new features into the technology. The latest Cilium release, v1.18 included major improvements for IPv6-only which were discussed with the maintainers at a KubeCon event only months before.
ESnet’s implementation proves that modern cloud native technologies can meet the unique requirements of scientific computing while maintaining the performance and reliability standards essential for advancing scientific discovery.
Key Impacts include:
- Consolidated hundreds of VM deployments into 2 multi-tenant production clusters.
- Achieved IPv6-only infrastructure supporting federal mandate compliance.
- Enabled unprecedented observability through Hubble integration for network debugging.
- Unified multiple networking components (CNI, load balancer, ingress, Gateway API) into a single tool.
- Implemented identity-based security using SPIFFE/SPIRE for fine-grained workload policies.
- Created a reusable blueprint for IPv6-only deployments shared across national laboratories.