Member post originally published on the Fairwinds blog by Joe Pelletier

The widespread adoption of containerized applications has fundamentally changed how organizations develop, deploy, and manage their software infrastructure. Kubernetes is fundamental to this change, because it makes it possible to manage containerized workloads and services at scale. One of the cornerstones of its effective operation is Kubernetes governance, the policies and procedures that govern how Kubernetes is configured, managed, and secured.

Without Kubernetes governance, platform teams are challenged with onboarding new applications and teams. Governance enables consistency, predictability, and repeatability — enabling platform teams to build “paved roads” to production so engineering teams use cloud-native infrastructure to deploy applications more frequently — a key metric for the DevOps Research and Assessment team (DORA).

What is Kubernetes Governance?

Kubernetes governance dictates the rules of operation in your Kubernetes environment, ensuring an aligned and controlled management of your Kubernetes clusters. It includes management of your Kubernetes resources, role-based access control, scheduling, and upgrades. Governance also includes the process for making decisions related to Kubernetes, including how to manage feature requests, security issues, and bug fixes. It’s important to have governance in place because Kubernetes introduces new technologies and different complexities for development, operations, and security teams. Without governance, challenges emerge, such as:

Kubernetes governance initiatives help ensure Kubernetes meets your organization’s policy requirements, adheres to best practices, and meets relevant regulatory requirements. 

Best Practices for Kubernetes Deployment 

To maximize the benefits of a Kubernetes implementation, follow these five best practices:

1. Security configurations

Must be established and enforced by your security team, ideally through automation and robust policies.

2. Cost optimization

Set resource requests and limits on workloads to maximize infrastructure utilization while still ensuring optimal application performance.

3. Reliability

Ensure workloads are configured with liveness probes and readiness probes, and follow Infrastructure as Code (IaC) best practices. IaC ensures infrastructure is auditable and consistent.

4. Policy enforcement

Once Kubernetes deployment increases beyond a single application, enforcing policy is critical. Tools and automation can help you prevent common misconfigurations and enable IT compliance. It can also promote a service ownership model because users are comfortable deploying, knowing that guardrails are in place to enforce the policies. One open source tool for cloud native environments is theOpen Policy Agent (OPA), which offers policy-based controls.

5. Monitoring and alerting

In an ephemeral environment like Kubernetes, it’s important to make sure that your infrastructure and applications are running. There are a number tools available to optimize monitoring.

Important Policies for Kubernetes Governance

Cluster-wide policies

Organizations can deploy both cluster-wide and namespace-specific (or application-specific) policies. Usually, cluster-wide policies apply to all workloads and may relate to security, efficiency, and reliability categories. A few important policies include: 

Namespace policies

Namespace-specific policies enforce standards for specific app teams or services when you need an increased level of security, efficiency, or reliability. You can use namespaces to create different ‘tenants’ within a shared Kubernetes cluster for teams; these teams must adhere to a common set of best practices that avoid disruption to other cluster tenants, such as resource exhaustion or security violations.

Enforcement points

Enforcement of policies can happen in multiple stages. Kubernetes governance enforces policies by delivering feedback to engineers in the tools they use, at the time they need it.

Cost allocation

Kubernetes spend increases proportionally based on the number of clusters, where apps and services are deployed, and how they are configured. Platform engineering teams need to allocate and showback costs in a business-relevant context to manage spend.

Resource labeling

Using namespaces or labels to map costs to a Kubernetes component helps you allocate costs to individual business units. The Kubernetes Vertical Pod Autoscaler (VPA) uses historical memory and CPU usage of workloads in conjunction with current pod usage to generate recommendations for resource requests and limits

Cost avoidance and optimization

Cost avoidance means reducing usage and optimizing costs to get a better cloud rate. Platform engineering teams can achieve these goals by shipping applications more rapidly, optimizing cloud usage, and reducing risks. 

Infrastructure as Code scanning

IaC enables you to use a configuration language to provision and manage infrastructure, applying the repeatability, transparency, and testing of modern software development to infrastructure management. IaC reduces error and configuration drift, allowing engineers to focus on work that contributes to larger business objectives. 

Container image scanning

A vulnerability in your base image exists in every container that contains that base image. Staying up to date with patches to your base image, controlling permissions, and eliminating images not required for deployment to production helps increase the security of your container images. 

Deprecated APIs

Kubernetes deprecates versions of APIs to minimize the need to maintain older APIs and push organizations to use more secure, up-to-date versions. When an application or service includesdeprecated or removed API versions, find and update them to the latest stable version.

Upgrade add-ons

Add-ons provide additional Kubernetes functionality. Sometimes add-ons require upgrades, but it’s important to check whether the latest version is compatible with your cluster. This process can be slow and difficult without tools that detect add-on changes automatically.

Policy enforcement and guardrails

In Kubernetes environments consisting of multiple Kubernetes clusters and teams of developers, having defined policies and an automated method of enforcing those policies is essential. These guardrails prevent deployment of changes that break something in production, allow a data breach, or enable configurations that do not scale properly.

Compliance analysis 

Most organizations must comply with multiple security and privacy regulations, such as SOC 2HIPAAISO27001GDPR, PIPEDA, and many more. Adopting defined Kubernetes compliance requirements policies and enforcing them across all clusters automatically is critical to achieving compliance goals, as well as automating compliance analysis to assess whether you are meeting changing requirements.

Implementing Kubernetes Governance

Kubernetes governance aligns with cloud native computing strategy, enabling platform teams to apply guardrails automatically that implement and enforce policies. These policies can help you implement and run Kubernetes reliably, securely, and cost-efficiently. This enables you to  maximize your investment in the K8s platform without worrying about whether you are meeting your organization’s policy requirements.