At KubeCon + CloudNativeCon Europe in Amsterdam from March 23-26, CNCF brought together a roundtable with experts in the cloud native ecosystem, including Ellis Tarn of AWS, Allan Naim of Google Cloud, Jorge Palma of Microsoft, and Nina Polshakova of solo.io. The discussion centered on how cloud native principles enable AI in production environments. The panelists shared key takeaways on the shift to AI-native computing, emphasizing that moving AI workloads into enterprise production requires three core components: a foundational, vendor-neutral infrastructure focused on platform maturity, integrated security for autonomous agents, and active community contribution.
How do we define production readiness for AI?
Organizations achieve production-readiness for AI when they meet a multi-dimensional standard of platform maturity. Panelists agreed the most important signal is alignment with the Kubernetes AI Conformance program, which identifies the essential primitives for serving and training AI at scale, guaranteeing interoperability across environments.
Readiness requires three key elements:
- Platform Maturity: This includes providing robust support for research scientists and Python users who need specialized environments.
- Security by Design: Security must be a priority from the start, particularly for agentic flows, ensuring agents operate within a secure, governed framework.
- Active Community Contribution: Organizations should move past simply consuming tools and actively help drive the next wave of innovation within CNCF Special Interest Groups (SIGs).
Why is scaling AI workloads a challenge?
Scaling AI workloads is significantly more difficult than scaling conventional microservices because AI workloads behave like enormous monoliths. This difficulty arises from the need to initialize multidimensional matrices in memory across numerous client nodes. Standard Kubernetes was not designed for the tight coupling required by these high-performance compute tasks.
What is the cloud native community doing to refactor Kubernetes for AI?
Engineers across the ecosystem are collaborating on key initiatives to evolve Kubernetes for high-performance compute without creating inflexible architectures. These efforts include:
- Pod Groups (Workload API): This initiative treats sets of pods as single failure domains, ensuring the proximity and reliability necessary for large-scale AI matrix initialization.
- Dynamic Resource Allocation (DRA): DRA integrates specialized chips and GPUs into the Kubernetes scheduler to manage hardware nuances and enable efficient AI training and serving.
- Inference Gateways: These utilize Gateway API standards to build efficient AI Gateways, which help with prompt management and high-intensity generative model responses.
How does AI change the role of an engineer?
AI is reshaping internal engineering roles. Prototyping has replaced the traditional Product Requirements Document (PRD), as product managers begin with AI-generated prototypes to test ideas before formal documentation. This shift, however, created a review bottleneck: the challenge is managing the sheer volume of generated code that requires human review. The panel suggested that the future moves toward agentic SRE, where AI agents assist with root-cause analysis and remediation while always keeping humans involved in mission-critical decisions.
Securing the AI supply chain
Security now extends beyond traditional container scans to focus on the integrity of the model supply chain and the risks associated with non-deterministic outputs. The community is focused on two main security efforts:
- Consistent Evaluation: Implementing consistent evaluation frameworks (Evals) and guardrails before models are deployed to production.
- Open Standards for Citation: Investment in community-driven controls is protecting against remote code execution via prompt injection. By adhering to open standards like llms.txt and standardized schema markups, the community ensures that any AI models crawling the web cite and recommend only authoritative, trusted open source sources.
The panelists concluded that when someone asks AI, “How do I scale this?” the answer should be rooted in open, interoperable, and vendor-neutral cloud native standards.