Preferred Networks

Preferred Networks unifies container networking for AI clusters with Cilium

Preferred Networks is one of Japan’s leading artificial intelligence companies, founded in Tokyo in 2014. It develops technology across the entire AI value chain, spanning AI chips, computing infrastructure, generative AI foundation models, and the solutions built on top of them for industries including manufacturing, life sciences, finance, and retail.

Preferred Networks has developed its own AI accelerator, MN-Core, to optimize energy efficiency. MN-3, a supercomputer powered by MN-Core, topped the Green500 list of the world’s most energy-efficient supercomputers three times.

Challenge

Today, Preferred Networks operates one cluster in each of its five locations. All clusters run on Kubernetes, with three on-premises and two on cloud IaaS, the largest holding more than 300 nodes. Each cluster combines several types of GPUs with the company’s in-house MN-Core accelerators (xPU). In total, the company runs over 1,500 GPUs and over 200 MN-Core accelerators to power its research and products. A team of roughly 30 designs, builds, and manages the infrastructure, split between a Cluster Infrastructure team that owns the physical layer and a Cluster Services team that owns the software.

Most of the clusters, the two cloud clusters and two of the three on-premises clusters, already ran Cilium. The challenge was in the final on-premises cluster which was running Calico. In the on-premises clusters, Preferred Networks’ switches used BGP to peer with the server hosts, and each server also ran its own BGP routing process. The aim was to let Pod traffic route directly over that network, but very few CNI plugins are built to work alongside an existing BGP process on the host.

“Calico has its own BGP process, and that created a direct conflict with the one already running on our hosts,” said Sho Shimizu, an engineering manager on the Cluster Services team at Preferred Networks. “We needed a CNI plugin that could work alongside our existing setup, not compete with it.”

The team was unable to make the two approaches coexist in this cluster, and fell back to tunneling, which wrapped Pod traffic in an extra layer and slowed networking down. Tunneling also made failures hard to trace. At one point, Pods lost the ability to resolve internal DNS through CoreDNS, and despite extensive investigation the team could not identify the cause. Maintaining two separate cluster architectures was not sustainable, so Preferred Networks decided to retire the old cluster entirely and build its replacement from scratch. Cilium, already proven across the rest of the fleet, was the clear choice for the new stack.

“For AI workloads, network performance directly affects how fast our models train and how efficiently we use our hardware,” said Shimizu. “Cilium offered the direct routing we needed, without the complexity of managing different networking stacks across our clusters.” 

Solution

Cilium was the only CNI plugin that fit all of the team’s needs. Its BGP Control Plane feature works alongside the host’s BGP process without conflicting with it, and Cilium also offers advanced Network Policy capabilities, along with detailed observability through Hubble.

In the on-premises clusters, Cilium peers with the BGP process on each host and advertises the addresses of the Pods on that node. That process passes the routes on to the datacenter switches, so the network can reach every Pod directly without tunneling. Shimizu and Yutaro Hayakawa walked through this design in their KubeCon + CloudNativeCon Japan 2025 talk, “BGP Peering Patterns for Kubernetes Networking at Preferred Networks.”

Before deploying to production, the team validated the design in a test environment. The main challenge was the number of custom resources Cilium required to configure BGP across a cluster, minimizing the number of custom resources needed to configure BGP across a large cluster.  The team found an approach where a single CiliumBGPPeeringPolicy could apply to every node. “We did not want our networking configuration to grow every time the cluster did,” said Shimizu. “Now adding a node never means touching a BGP custom resource, even on a cluster with more than 300 nodes.” 

Impact

Standardizing on Cilium simplified the company’s networking. The workarounds the team had relied on with Calico were no longer needed, and eliminating tunneling improved both performance and visibility. The DNS connectivity issue has not recurred, and networking has run reliably since.

Cilium’s Network Policy capabilities go well beyond standard Kubernetes network policy, giving the team more flexibility to isolate tenants across clusters. Hubble provides flow-level observability, making connectivity issues significantly easier to diagnose.The consolidation was completed in April 2026, when the last Calico-based cluster was shut down. Preferred Networks now runs Cilium across all five clusters, with the on-premises clusters leveraging BGP and the cloud clusters using VXLAN, since the cloud provider does not allow BGP peering. 

“Networking is the foundation of a Kubernetes cluster, so its stability matters more than almost anything else,” said Shimizu. “Since we moved to Cilium, we haven’t had to think about that layer, and running the same plugin on-premises and in the cloud means we maintain one stack instead of several.”

What’s next: CRD migration and further Cilium adoption

Preferred Networks has upgraded Cilium regularly since adopting it, starting with v1.13.3 and running v1.18.9 today. The next step is upgrading to v1.19.x, which removes the older BGP configuration format the team had relied on. The team has already migrated to the current format to clear that path, and expects to complete the v1.19.x upgrade later this summer.

Beyond that, the team is considering several improvements, including adopting Cilium’s kube-proxy replacement, using Cilium in place of MetalLB to advertise load balancer addresses, and making more use of the metrics Cilium exports for observability.