Executive Summary
As demand for AI compute surges, a single urban data center is approaching a hard physical limit: power and land. Instead of expanding one data center, NTT Docomo Business distributed GPUs across many DCs. We built GPU over APN Testbed,, which operates eight data centers across Japan as a single Kubernetes cluster. Spanning more than 2,000 km from Sapporo in the north to Fukuoka in the southwest, the eight sites are tied into a single L2 domain by a 100 Gbps, low-latency, high-capacity L2 network over IOWN APN. The entire fabric sits within Japan; GPUs are delivered through Petasus AI Cloud — SK Telecom’s KubeVirt-based GPU virtualization software, deployed on the Testbed — so we run a fleet of heterogeneous GPUs, configured differently at each site, as a securely isolated, multi-tenant single cluster, and we achieve wire-rate GPU-to-GPU communication between Tokyo and Osaka.
Company Overview
NTT DOCOMO BUSINESS, Inc. (formerly NTT Communications) is a leading Japanese telecommunications operator that provides the ICT platforms behind the digital transformation and AI infrastructure of enterprises and regions. We aim to deliver AI-native infrastructure — reaching all the way to the edge — under a cloud native operating model, optimizing resources such as GPU, network, and power. GPU over APN Testbed is one of those efforts.
Projects used
By the numbers
8 DCs, 2,000+ km:
Eight data centers from Sapporo to Fukuoka operated as a single Kubernetes cluster — a GPU fabric that power and land constraints would otherwise confine to one site, distributed across many.
5 business days → tens of minutes:
Shifting from the originally planned bare-metal leasing to KubeVirt VM delivery dramatically shortened environment provisioning and cleanup.
100 Gbps wire rate:
Tuned NCCL all-reduce between Tokyo and Osaka, inside KubeVirt GPU VMs, hit the 100 Gbps line (busbw ~12.9 GB/s).
Challenge: Power, Land, and a Single-Cluster Requirement
As AI models scale, so does demand for GPU clusters — and for power. A single urban data center is nearing its limits on power and land, and this constraint cannot be resolved by adding racks. Large-scale GPU fabrics have therefore had to concentrate in a single site with power and land to spare.
The Testbed’s goal was to distribute GPUs across data centers while preserving the usability of a single site: multiple tenants coexisting safely, provisioning on demand, and one uniform cloud native operating model across all eight locations. That goal led to three requirements:
- Customers must bring their existing training and inference workloads as-is, without modification.
- Multiple tenants must be provided a securely isolated, wide-area GPU fabric, quickly.
- The round-trip time (RTT) that comes with distance between sites cannot be eliminated, so it must be designed in as a constraint.
The first requirement shaped the architecture. If each site were its own cluster, workloads that span sites would require rewriting for multiple clusters or an added cross-cluster mechanism. To avoid workload modification, we operate all eight sites as a single Kubernetes cluster and hand over compute resources with GPUs already attached, so existing workloads run without rewriting.
The second requirement came from how we onboard tenants. To accept multiple tenants onto shared infrastructure, secure isolation was a must — and provisioning had to be fast. Bare-metal leasing was expected to take about five business days per tenant to set up and clean up, and could not share a single node across tenants. Plain Kubernetes containers, meanwhile, share the host kernel, so namespace and network isolation alone are not strong enough for secure multi-tenant separation. We needed on-demand provisioning and secure tenant isolation together.
Solution: A Wide-Area GPU Fabric Realized with KubeVirt
NTT Docomo Business builds and operates the GPU over APN Testbed, our wide-area fabric. Using IOWN APN’s 100 Gbps L2, we unify eight sites (four in the Tokyo metropolitan area, plus Osaka, Sapporo, Fukuoka, and Kanazawa) into a single L2 domain, so heterogeneous GPUs across the country can be treated as one pool.
Figure: GPU over APN Testbed’s eight data centers, tied into a single wide-area L2 domain over IOWN APN — from Sapporo to Fukuoka.
On top of that, what provisions GPUs as virtual machines is SK Telecom’s Petasus AI Cloud — a KubeVirt-based GPU virtualization stack that also supports NVLink use inside the VM, something stock KubeVirt does not offer. We realized Petasus on the Testbed in collaboration with SK Telecom.
Why KubeVirt — Not Bare Metal, Not Plain Containers
KubeVirt satisfies the requirements above — no workload modification, on-demand provisioning, and secure tenant isolation — together:
- Provisioning went from five business days to tens of minutes. Starting, stopping, and cleaning up VM workers completes on demand, greatly reducing the effort of providing environments.
- GPU nodes went from “dedicated” to “shared.” With bare metal, even when only a few GPUs were needed, one tenant had to occupy an entire node. With VMs, multiple tenants share the same physical node, each VM assigned its own GPUs.
- Performance is almost the same as bare metal. Both GPUs and NICs attach directly to the VM through VFIO-PCI passthrough. But passthrough alone does not guarantee performance: Petasus schedules GPUs and RDMA NICs so that their NUMA and PCIe topology inside the VM mirrors the physical layout, preserves NVLink connectivity across passed-through GPUs, and carries GPUDirect RDMA across the VM boundary — which is why the performance described below was achievable inside VMs.
- Cloud native lifecycle management, such as VM snapshots and migration, works as-is on the GPU platform.
- Isolation is enforced at the hardware boundary, not the kernel boundary. Each tenant runs inside its own VM with a dedicated guest kernel, its own passed-through GPUs, and dedicated SR-IOV virtual functions on the RDMA fabric — so tenants sharing a physical node never share a kernel, a device, or a network path. This gave us the confidence in multi-tenant isolation that container namespaces alone could not.
There was, however, one unavoidable condition when provisioning GPUs as VMs: NVLink had to be usable inside the VM. If GPUs cannot connect over NVLink within a VM, multi-GPU workloads cannot deliver their true performance — and then there would be little point in having built the Testbed with high-end GPUs and IOWN APN. Yet stock KubeVirt does not yet support in-VM NVLink. The Petasus AI Cloud that makes this possible was indispensable for us. Built on KubeVirt, it goes as far as aligning GPU and NIC topology to the underlying NUMA/PCIe layout, and carrying both GPUDirect RDMA and GPUDirect Storage (GDS) across the VM boundary.
“Passing a GPU into a VM is easy; making the workload unable to tell it apart from bare metal is the hard part. Petasus AI Cloud presents GPUs, NVLink, and RDMA NICs to the guest exactly as the hardware intends — topology, NUMA alignment, GPUDirect and all. Watching NCCL run at wire rate between Tokyo and Osaka, inside VMs, was the moment we knew GPU virtualization was ready for wide-area scale. And it was Kubernetes and KubeVirt as a common language that let engineers in Seoul and Tokyo get there together,” Jian Li, Principal Engineer at SK Telecom and the architect and lead developer of Petasus AI Cloud.
Operating Eight Sites Nationwide as One Cluster
That the Testbed is a single Kubernetes cluster is a major operational advantage in itself. The Kubernetes control plane, including etcd, resides in the Tokyo metropolitan area, while the nodes distributed to regional sites are workers. Nodes scattered across eight sites nationwide are handled together as one platform, so we never take on the complexity of managing fragmented, separate clusters. Even with a mix of heterogeneous GPUs — B200, H200, H100, RTX PRO 6000, L40S — and AI storage including Lustre (DDN EXAScaler), we can operate them without trouble.
The Testbed’s architecture does not depend on a specific transport technology; it needs only a low-latency, high-capacity L2. We realized this wide-area L2 with IOWN APN. Open APN is an open specification published through the IOWN Global Forum, so as long as there is an equivalent L2, this architecture can be reproduced in other environments.
Open Source Brought Us Together — Cross-Border Co-Creation
It is also worth mentioning that this collaboration began with open source. It was precisely because Kubernetes and KubeVirt provided a shared foundation that engineers at NTT Docomo Business and SK Telecom could discuss the same design across national borders. The division of work — the wide-area L2 fabric and cluster operations (NTT Docomo Business), and the GPU virtualization stack including in-VM NVLink (SK Telecom / Petasus) — rests on that common base. Neither company alone, we believe, would have arrived at this architecture.
Impact: Measured Performance on VMs, Across the Fabric
To measure how distribution and virtualization affect performance, we ran every measurement inside the KubeVirt GPU VMs. The results:
- GPU-to-GPU communication (NCCL all-reduce): with tuned NCCL settings, all-reduce between Tokyo and Osaka reached the 100 Gbps wire rate — busbw 12.9 GB/s out-of-place, 12.1 GB/s in-place (≒ 100 Gbps) [1].
- Inter-site storage (DDN EXAScaler / Lustre over RDMA):
Measured 11.3 GB/s write / 10.1 GB/s read [2] between Tokyo (Akihabara) and Osaka, and 11.9 GB/s write / 11.7 GB/s read over the much shorter Tokyo–Yokohama span within the metro area. Despite the vast difference in distance, the storage throughput remains effectively unchanged and consistently high.
Note 1: busbw is a normalized metric NCCL uses so results can be compared directly against the physical link. It can appear to exceed the nominal capacity (100 Gbps ≈ 12.5 GB/s); this does not mean wire-rate was exceeded — actual link utilization stayed within the physical limit.
Note 2: Results based on per-direction tuning. With a common configuration, Tokyo–Osaka performance is 11.3 GB/s write / 7.8 GB/s read.
These results show that distribution, virtualization, and distance do not compromise practical performance for the targeted workloads.
At KubeCon NA 2025, in-VM NVLink support has been discussed as a direction the KubeVirt project is heading. We already run it in production — at a nationwide scale. By standing up a securely isolated, multi-tenant single cluster across data centers spanning all of Japan, we have shown that GPU infrastructure can sit wherever power and land are available. For platform operators facing the limits of a single urban data center, we believe this can be one solution.
“What we value most is that this architecture was created through true co-creation. By building on a common foundation of CNCF technologies, engineers from different companies and different countries were able to contribute as equals, bringing together their respective strengths to create something that neither organization could likely have achieved on its own,” Teruaki Noyama, Senior Manager at NTT Docomo Business.
The Honest Trade-offs, and a Testbed for Real Workloads
Of course, there are honest difficulties too. You cannot beat the physics of signal speed, so the longer the distance, the larger the RTT. That makes it a prerequisite to judge which workloads are suited to distribution and to place them with RTT in mind.
GPU over APN Testbed is not a closed product; it is a testbed for validating wide-area distributed GPU operations in a real environment. Customers can bring questions like “does this workload hold up under wide-area distribution?” and measure the answer. The Testbed’s purpose is to give anyone facing the same power and land constraints a place to validate the reach of a wide-area distributed GPU fabric in a real environment.
“We don’t see GPU over APN Testbed as a finished answer. Whether you offer distributed GPUs to others or run them yourself, which workloads suit wide-area distribution is a question we want to work through together — and we’ll keep evolving the platform, in hardware, network, and software, as we go,” said Xiaojing Zhang, Director, NTT Docomo Business.
Our biggest lesson was simple: the latency that comes with distance cannot be eliminated, but when workloads are placed to match their characteristics, practical performance can be drawn out.
Future Plans — Farther, and More Dynamic
To advance the Testbed, and the community, we plan to work on the following:
- Adopting DRA (Dynamic Resource Allocation) for dynamic GPU orchestration beyond static resource limits.
- As an open testbed, trying wide-area GPU fabrics together with customers, co-developing new use cases, and expanding sites and nodes accordingly.
- Giving back to the Kubernetes and KubeVirt communities — including a joint NTT Docomo Business / SK Telecom talk at KubeCon + CloudNativeCon Japan 2026.
For any organization facing the same power and land constraints, we hope this Testbed configuration can be a starting point. With an open, low-latency, high-capacity L2 like IOWN APN, and with KubeVirt, the same should be achievable elsewhere.