[meta] Explore a complete MLOps workflow using KitOps for model packaging, Cog for local testing, and KAITO for scalable Kubernetes deployment.
For ML engineers and AI app developers, there can be several bottlenecks in the end-to-end MLOps pipeline that can block your solutions from shipping quickly. These barriers range from variability across environments, to limited compute resources, to model or data versioning. While Kubernetes offers basic tooling and frameworks to address some pain points, there are several open source tools that make it even easier, enabling users to integrate AI faster in their applications. In this post we explore the intersection of 3 key open source projects: KitOps, Cog, and KAITO, to help you get from ML code to production on Kubernetes in minutes. So, what does each of these projects accomplish?
KitOps, a CNCF Sandbox project, was designed to bring security, control, and ease-of-use to the AI/ML supply chain. It standardizes how AI projects are organized, versioned, and stored. By adapting the existing OCI standard to work with AI/ML, KitOps-packaged AI/ML projects can be easily used by any tool that supports containers. Store KitOps ModelKits in your container registry, pull them into your existing pipelines, sign them with your container signer, and attach attestations. You can use KitOps to manage all the artifacts and dependencies for any AI/ML project you’re developing.
Cog is a tool that standardizes the setup/testing of AI projects and provides boilerplate code to write inferencing calls against local models. Before provisioning the infrastructure for AI in production, Cog can be used to define the prediction code, build the Cog container and test the model locally to save time and compute resources.
KAITO is a CNCF Sandbox project that automates the deployment of the model inference service and underlying compute in a production-ready environment. To support the high availability of your AI workloads, KAITO enforces smart defaults in your LLM serving engine to prevent out-of-memory errors and avoid painful workload interruptions!
Why These Tools Work Well for MLOps
KitOps, Cog, and KAITO aren’t just three random tools—they’re designed to solve complementary problems in the MLOps workflow, creating a seamless handoff between development phases.
The ModelKit you create with KitOps, test with Cog, and deploy with KAITO is the exact same artifact, no translation, no reformatting, no “it works on my machine” surprises.
Traditional MLOps workflows suffer from disconnected stages. You might containerize your model one way for local testing, package it differently for storage, and deploy it with yet another approach for production. Each transition introduces potential failures, security risks, attack vectors, and inconsistencies.
These tools eliminate those risks and share a common philosophy around containerization and standardization:
KitOps packages your entire ML project (model, code, data, configs) into a single, versioned artifact using OCI standards (the same container technology that powers modern software deployment).
Cog consumes this standardized package and provides a consistent local testing environment, ensuring what you test locally matches what will run in production.
KAITO takes the same containerized artifacts and intelligently provisions the right infrastructure, applying production-ready configurations that prevent common deployment failures.
Because all three tools embrace container standards, your ML project becomes truly portable and can use the processes and tools you’ve proven work with your other containerized applications.
This eliminates the typical MLOps bottlenecks: environment drift, manual configuration errors, and the expensive cycle of discovering problems only after provisioning production infrastructure.
By using KitOps, Cog, and KAITO you can organize and standardize critical MLOps tasks like:
- Easily and consistently benchmarking the performance of different models in the same environment and infrastructure
- Identifying errors or bottlenecks early in model validation, before provisioning costly compute resources
- Employing security best practices with an OCI-compliant registry by default to securely manage and distribute containerized model artifacts
- Creating repeatable AI deployment patterns across teams
- Introducing automation to create feedback loops across the major stages of MLOps
Get started with KitOps, KAITO, and Cog
Now it’s your turn. You can install KitOps from https://github.com/kitops-ml/kitops, Cog from https://github.com/replicate/cog , and KAITO from https://github.com/kaito-project/kaito.
Once you’ve done that we encourage you to check out this sample walkthrough that we created on Azure Kubernetes Service (AKS). The workshop explores how to deploy open-source models on Azure Kubernetes Service (AKS) using KAITO, KitOps, and Cog to build production-ready containers for model inference. By the end of it, you will be able to:
- Deploy an AKS cluster with KAITO installed and attach Azure Container Registry (ACR) with Terraform CLI
- Bootstrap a new ModelKit project by downloading an open-source model from HuggingFace with Kit CLI
- Create a model inference project with Cog CLI
- Build and push Cog app container image to ACR with Cog CLI
- Pack and push ModelKit (model and inference code) to ACR with Kit CLI
- Self-host Cog model inferencing application on AKS with KAITO
We hope you find value in this combination of open source tools. The next time you’re staring at a model that works locally but won’t deploy, remember: it doesn’t have to take weeks. With this integrated approach using KitOps, Cog, and KAITO, you can go from experimentation to production-ready deployment in the same day.
Try our complete AKS sample and see the difference for yourself. Then share your results with the community—we’d love to hear how these tools are accelerating your AI deployments. For support, release updates, and general KitOps discussion, please join the KitOps Discord, and follow KitOps on X for daily updates.