Seres

How SeresCloud Scaled AI Inference to 10 Billion Daily Tokens with Kubernetes and HAMi

Challenge

2026 is widely regarded as the inaugural year of AI Agents. Inside Seres, the adoption of intelligent agents and model token consumption have seen explosive growth. One telling metric: the number of online model types running on the SeresCloud platform grew from around 10 at the start of the year to 42+ today, while the daily token volume processed for internal businesses climbed from 300 million to 10 billion+ — an increase of more than 3000%.

Figure 1: SeresCloud business growth trends (daily token requests and online model count, Jan–Jul 2026)

Growth at this scale posed serious challenges for our AI infrastructure. On the GPU resource management side, we initially adopted NVIDIA’s official GPU-Operator solution. It can allocate whole GPUs into pods on demand, and for MIG (Multi-Instance GPU)-capable hardware (such as A100/H100) it can also partition a card at the physical level, splitting one GPU into multiple slices for different pods.

However, as the cluster scaled up and GPU models diversified, the limitations of GPU-Operator gradually surfaced:

This led us to the open-source HAMi. HAMi supports GPU memory and compute slicing down to a granularity as fine as 1%, offers a unified scheduling interface across 11+ heterogeneous devices including NVIDIA, Huawei Ascend, Hygon DCU, and Cambricon MLU, and is an active CNCF Incubating project — making it an excellent fit for our needs.

Even before adopting HAMi, we had built a complete inference compute platform on top of the SeresCloud cloud platform. It exposes a unified model-service interface to internal users and agent applications, and interfaces with Kubernetes clusters and GPU resource pools underneath. The overall architecture:

Solution

Core platform capabilities include:

• Model repository: manages versions, images, and configurations for 40+ models, with one-click deployment
• Inference gateway: a unified entry point handling traffic routing, load balancing, and rate limiting / circuit breaking
• Elastic scaling: performs HPA-style horizontal scaling automatically based on real-time QPS, GPU utilization, and queue depth
• Real-time monitoring: end-to-end observability over inference services, GPU nodes, and memory/compute usage

HAMi sits as the GPU virtualization and scheduling layer at the bottom of this architecture, complementing our in-house platform: the platform handles elasticity at the business layer, while HAMi handles fine-grained scheduling and isolation at the resource layer.

Engineering Practice

A Leap in GPU Utilization for Small-Model Inference

We provide internal users with a large number of lightweight model services, including Embedding, Rerank, speech synthesis (Voice), and OCR. These services share a few typical characteristics:

• Small model sizes (from a few hundred MB to a few GB)
• Low per-inference memory footprint (typically 1–4 GB)
• Markedly tidal request patterns (peak during the day, trough at night)

During the GPU-Operator exclusive-mode period, GPU utilization in these scenarios stayed roughly between 15% and 25%. We also tried preemptive sharing schemes such as Time-Slicing — where multiple pods share time slices of the same GPU. This did improve utilization, but without hard isolation it carried the following risks:

• A memory spike in one pod could cause OOM in other pods sharing the same card
• High-load pods could monopolize compute, causing latency jitter for lower-priority services
• Large blast radius: a single pod crash on a card could affect the entire card

Ultimately, we adopted the HAMi vGPU soft-isolation sharing mode across these scenarios. By intercepting CUDA calls inside the container via libvgpu.so, HAMi enforces hard limits on both memory and compute:

resources:
  limits:
    nvidia.com/gpu: 1          # Request 1 vGPU share of a physical GPU
    nvidia.com/gpumem: 3000    # Limit memory to 3000 MiB
    nvidia.com/gpucores: 30    # Limit compute to 30%

Combined with the SeresCloud platform’s dynamic scaling — automatically scaling in to release resources when idle and scaling out fast to absorb traffic when busy — GPU utilization in our small-model inference scenarios has steadily risen to around 90%.

Choosing a Scheduling Strategy: Binpack vs Spread

HAMi offers multiple scheduling strategies; we primarily use two in production:

By assigning different scheduling strategies to different workloads, we improve utilization while protecting the SLAs of business-critical services.

Key Production Configurations

Below are the core Helm configurations (excerpt) from our HAMi deployment:

devicePlugin:
  deviceSplitCount: 10        # Max 10 shared tasks per physical GPU
  deviceMemoryScaling: 1      # Memory based on actual physical VRAM (no oversubscription)
  deviceCoreScaling: 1        # Compute based on actual physical compute
  migStrategy: none           # MIG disabled (handled by HAMi soft slicing)
  disablecorelimit: "false"   # Enable compute limiting

scheduler:
  defaultMem: 5000            # Default memory allocation 5000 MiB
  defaultCores: 0             # No compute reserved by default (dynamically allocated by the scheduler)

deviceSplitCount=10 is a key concurrency cap. It defines the maximum number of vGPU tasks that can run concurrently on a single GPU, but it does not change the total amount of physical memory or compute — the scheduler still performs admission control based on actual remaining resources, ensuring that oversubscription never causes runtime OOM.

Impact – Results and Benefits

Lessons Learned

While productionizing HAMi, we ran into a few experiences worth sharing:

1. Webhook compatibility with privileged pods: HAMi’s Mutating Webhook automatically rewrites the schedulerName of any pod requesting vGPU resources to hami-scheduler. For debugging pods that require privileged: true, the webhook skips processing — something to keep in mind during troubleshooting.
2. Observability of scheduling failures: From v2.x onward, HAMi greatly improved the output of scheduling-failure reasons, including standardized error codes such as NodeInsufficientDevice, CardTypeMismatch, and CardInsufficientMemory. We recommend integrating HAMi’s metrics into Prometheus/Grafana to pinpoint scheduling issues quickly.
3. Tuning deviceSplitCount: Bigger is not always better. Excessively high concurrency increases context-switching overhead and operational complexity. After load testing, we settled on deviceSplitCount=10 for inference scenarios, striking a balance between density and stability.
4. Coordinating with in-house autoscaling: HAMi decides “which vGPU on which card a pod lands on”, while our in-house platform decides “when to add or remove pod replicas”. The key to making them work together is unifying monitoring metrics — we fuse the vgpu_monitor metrics exposed by HAMi with the platform’s own QPS/latency metrics as input for scaling decisions.

Future Plans

1. Expanding the heterogeneous compute pool: As the heterogeneous AI accelerator ecosystem matures, we plan to bring devices such as Huawei Ascend and Hygon DCU into HAMi’s unified scheduling framework to build a truly heterogeneous compute pool.
2. HAMi-DRA adaptation: We are tracking the HAMi community’s progress on the Kubernetes DRA (Dynamic Resource Allocation) standard, aiming to enable more flexible resource allocation through the native DRA interface in the future.
3. Fine-grained scheduling policies: We will explore advanced features such as priority-based resource preemption and GPU-topology-aware scheduling to further improve resource efficiency for large-model inference.
4. Giving back to the community: We will continue to feed the issues we encounter in production and our improvement suggestions back to the HAMi community, growing together with it.