Guest post, originally published by Gianluca Arbezzano.

Kubernetes runs a set of controllers to keep matching the current state of a resource with its desired state. It can be a Pod, Service or whatever is possible to control via Kubernetes. K8S has as core value extendibility to empower operators and applications to expand its set of capabilities. An event-based architecture where everything that matters get converted to an event that can be trigger custom code.

When I think about a problem I have that requires to take action when Kubernetes does something my first target is one of the events that it triggers, example:

• New Pod Created
• New Node Joined
• Service Removed and many, many more.

To stay informed about when these events get triggered you can use a primitive exposed by Kubernetes and the client-go called SharedInformer, inside the cache package. Let’s see how it works in practice.

First of all as every application that interacts with Kubernetes you need to build a client:

// import "os"
// import  corev1 "k8s.io/api/core/v1"
// import  "k8s.io/client-go/kubernetes"
// import  "k8s.io/client-go/tools/clientcmd"


// Set the kubernetes config file path as environment variable
kubeconfig := os.Getenv("KUBECONFIG")

// Create the client configuration
config, err := clientcmd.BuildConfigFromFlags("", kubeconfig)
if err != nil {
    logger.Panic(err.Error())
    os.Exit(1)
}

// Create the client
clientset, err := kubernetes.NewForConfig(config)
if err != nil {
    logger.Panic(err.Error())
    os.Exit(1)
}

As you can see I am commenting the code almost line by line to give you a good understanding about what is going. Now that you have the client we can create the SharedInformerFactory. A shared informer listens to a specific resource; the factory helps you to create the one you need. For this example it lookup the Pod SharedInformer:

 // import v1 "k8s.io/api/core/v1"
 // import "k8s.io/client-go/informers"
// import  "k8s.io/client-go/tools/cache"
// import "k8s.io/apimachinery/pkg/util/runtime"

// Create the shared informer factory and use the client to connect to
// Kubernetes
factory := informers.NewSharedInformerFactory(clientset, 0)

// Get the informer for the right resource, in this case a Pod
informer := factory.Core().V1().Pods().Informer()

// Create a channel to stops the shared informer gracefully
stopper := make(chan struct{})
defer close(stopper)

// Kubernetes serves an utility to handle API crashes
defer runtime.HandleCrash()

// This is the part where your custom code gets triggered based on the
// event that the shared informer catches
informer.AddEventHandler(cache.ResourceEventHandlerFuncs{
    // When a new pod gets created
    AddFunc:    func(obj interface{}) { panic("not implemented") },
    // When a pod gets updated
    UpdateFunc: func(interface{}, interface{}) { panic("not implemented") },
    // When a pod gets deleted
    DeleteFunc: func(interface{}) { panic("not implemented") },
})

// You need to start the informer, in my case, it runs in the background
go informer.Run(stopper)

Knowing about Shared Informers gives you the ability to extend Kubernetes quickly. As you can see it is not a significant amount of code, the interfaces are pretty clear.

Use cases

I used them a lot to write dirty hack but also to complete automation gab a system for example:

1. We used to have a very annoying error during the creation of a Pod with a persistent volume. It was not a high rate error a restart makes everything to work as expected. A dirty hack is pretty clear; I automated the manual process of restarting the pod with that error using a Shared Informer just like to one I showed you
2. I am using AWS, and I would like to push some EC2 tags down as kubelet labels. I use a shared informer but this time to watch when a new node joins the cluster. From the new node I can get its AWS instanceID (it is a label itself), and with the AWS API. I can retrieve its tags to identify how to edit the node itself via Kubernetes API. Everything is part of the AddFunc in the shared informer itself.

Complete Example

This example is a function go program that logs when a new node that contains a particular tag joins the cluster:

package main

import (
    "fmt"
    "log"
    "os"

    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/util/runtime"

    "k8s.io/client-go/informers"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/tools/cache"
    "k8s.io/client-go/tools/clientcmd"
)

const (
    // K8S_LABEL_AWS_REGION is the key name to retrieve the region from a
    // Node that runs on AWS.
    K8S_LABEL_AWS_REGION = "failure-domain.beta.kubernetes.io/region"
)

func main() {
    log.Print("Shared Informer app started")
    kubeconfig := os.Getenv("KUBECONFIG")
    config, err := clientcmd.BuildConfigFromFlags("", kubeconfig)
    if err != nil {
        log.Panic(err.Error())
    }
    clientset, err := kubernetes.NewForConfig(config)
    if err != nil {
        log.Panic(err.Error())
    }

    factory := informers.NewSharedInformerFactory(clientset, 0)
    informer := factory.Core().V1().Nodes().Informer()
    stopper := make(chan struct{})
    defer close(stopper)
    defer runtime.HandleCrash()
    informer.AddEventHandler(cache.ResourceEventHandlerFuncs{
        AddFunc: onAdd,
    })
    go informer.Run(stopper)
    if !cache.WaitForCacheSync(stopper, informer.HasSynced) {
        runtime.HandleError(fmt.Errorf("Timed out waiting for caches to sync"))
        return
    }
    <-stopper
}

// onAdd is the function executed when the kubernetes informer notified the
// presence of a new kubernetes node in the cluster
func onAdd(obj interface{}) {
    // Cast the obj as node
    node := obj.(*corev1.Node)
    _, ok := node.GetLabels()[K8S_LABEL_AWS_REGION]
    if ok {
        fmt.Printf("It has the label!")
    }
}

The next time you book a campsite or activity at a U.S. National Park, thank Kubernetes for making it quick and easy. 

As part of the White House’s IT modernization effort, Booz Allen Hamilton’s Strategic Innovation Group started working with the federal government to “provide a better experience to citizens in every way that we interact with the government through every channel,” says Senior Lead Technologist Martin Folkoff. 

One of the first tasks was to relaunch the decade-old recreation.gov website, which provides information and real-time booking for more than 100,000 campsites and facilities on federal lands across the country.

The infrastructure needed to be agile, reliable, and scalable—as well as repeatable for the other federal agencies that are among Booz Allen Hamilton’s customers. “The only way that we thought we could be successful with this problem across all the different agencies is to create a microservice architecture, so that we could be very dynamic and very agile to any given agency for whatever requirements that they may have,” says Folkoff. 

Booz Allen Hamilton, which has provided consulting services to the federal government for more than a century, introduced microservices, Docker containers, and AWS to its federal agency clients about five years ago. The next logical step was Kubernetes for orchestration. “Knowing that we had to be really agile and really reliable and scalable, we felt that the only technology that we know that can enable those kinds of things are the ones the CNCF provides,” Folkoff says. “One of the things that is always important for the government is to make sure that the things that we build really endure. Using technology that is supported across multiple different companies and has strong governance gives people a lot of confidence.”

In addition to its work with the Department of Interior for recreation.gov, Booz Allen Hamilton has brought Kubernetes to various Defense, Intelligence, and civilian agencies. Says Chief Technologist Josh Boyd: “When there’s a regulatory change in an agency, or a legislative change in Congress, or an executive order that changes the way you do business, how do I deploy that and get that out to the people who need it rapidly? At the end of the day, that’s the problem we’re trying to help the government solve with tools like Kubernetes.”

For recreation.gov, the impact was clear and immediate. With the Kubernetes platform, Folkoff says, “if a new requirement for a permit comes out, we have the ability to design and develop and implement that completely independently of reserving a campsite. It provides a much better experience to users.” Today, changes can be implemented in about 30 minutes, compared to the multiple hours or even days legacy government applications require to review the code, get approval, and deploy the fix. Developers can create and publish new services to production within one week. 

Additionally, Folkoff says, “supporting the large, existing monoliths in the government is extremely expensive,” and migrating into a more modern platform has resulted in perhaps 50% cost savings.

Read more about how Booz Allen Hamilton is helping the federal government modernize with Kubernetes in the full case study. You can also read the Wall Street Journal article!

Guest post by Anil Kumar, Director of product management, Couchbase

Digital transformation is radically reshaping the way organisations across the globe do business. Empowered by DevOps practices, IT teams are helping to drive down costs, enhance agility and create a new era of innovation-fuelled growth. But what drives DevOps? Increasingly, the answer is containers: viewed by many as a major evolution in cloud computing, providing scalability and flexibility where developers need it most. Yet for the enterprise architects tasked with maintaining IT infrastructure, the “dream” of containers can very quickly turn into a nightmare.

Container sprawl and interoperability issues with legacy technology including centralised databases threaten to undermine the DevOps project, and with it, the digital transformation efforts now so vital to business growth.

The beauty of containers

Containers could be described as the modern building blocks of cloud computing. Like virtual machines (VMs), they provide a neat, self-contained package in which developers can run their applications, libraries and other dependencies. In so doing, containers offer a consistent, predictable environment isolated from other applications. However, they’re more lightweight and have lower associated overheads than virtual machines, enabling them to be deployed quickly and easily at large scale across private, public and hybrid cloud environments.

It’s therefore no surprise to see why containers have garnered such positive press over recent years. The ability to set-up test environments quickly and easily and scale them up to full production if necessary is a tantalising prospect for developers. It’s claimed that over 80% of IT teams used containers in 2018, up from just over half (58%) the year before. Google alone says it starts over two billion containers each week.

From dream to nightmare

However, the rapid adoption of containers is making clear a growing rift in IT architecture, between stateless application workloads running on container environments, and stateful application workloads running on more traditional infrastructure. As container orchestration tools such as Kubernetes have allowed organisations to take greater control over their container environments, so businesses have begun to see the benefits of stateless applications – from allowing an online-first approach to services, to easier scalability and redeployment, and the ability to connect multiple applications into services using APIs.

Yet as organisations have taken full advantage of containers, they are now facing the opposite challenge from their legacy IT. Quite simply, architecture built for stateful applications cannot match the flexibility, agility and rapid evolution that is now possible. For instance stateful applications will often exist in silos, with their own independent network, policies and infrastructure – meaning it is much harder to scale without directly adding to that infrastructure, or to connect with other applications using APIs. What this means is that architects face an all-too-common nightmare of running without actually moving, as despite investment and energy put into building and improving legacy applications and their databases, the potential of stateless applications continues to accelerate over the horizon.

It’s clear that architects need to bridge this gap – as the longer they leave it, the wider and harder to cross it will become. The task will have to be delicate. The new lightweight approach containers allow is at odds with the traditional, monolithic approach of legacy databases and infrastructure. At the same time, simply replacing a legacy database with a more modern alternative is not an easy answer. That database will doubtless support applications that are absolutely critical to the business, and there is no guarantee that a more modern NoSQL database will automatically support containers.

Orchestrating DevOps success

The good news is that there is light at the end of the tunnel. Modern databases are being designed to operate seamlessly with new container orchestration tools like Kubernetes – allowing architects to more easily manage how containers connect with centralised databases in the cloud. With these tools to hand, architects can finally take a holistic approach to IT infrastructure, ensuring each component works well together.

The challenge for architects will be understanding which of their applications need to be moved from stateful to stateless quickly, to ensure they can keep pace with the evolution of containers; and which can be kept in their legacy environment, as they are at no risk of becoming obsolete. For instance finance and payment functions, whose prime concern is performing the exact same action consistently, quickly and transparently, could remain on their legacy database, while anything that impacts the customer or end user experience should be modernised so that it can keep evolving at the same rate as customer demands. Over time, almost all of the applications in a business will be built on containers. If they can manage this evolution, architects can ensure that containers remain both a DevOps dream and an architect’s best friend.

IT architects have an increasingly challenging role in the organisation, as they are not only tasked with keeping the lights on, but also providing the right environment to drive innovation-fuelled success. Containers are just the latest advances in technology that tests their ability to keep pace with DevOps teams. There will surely be more ahead. To keep adding value to the business, architects must continue to evaluate ways to integrate existing and emerging technologies.

Guest post by Mohamed Ahmed, originally published on the Magalix Blog

An excellent cloud-native application design should declare any specific resources that it needs to operate correctly. Kubernetes uses those requirements to make the most efficient decisions to ensure maximum performance and availability of the application.

Additionally, knowing the application requirements firsthand allows you to make cost-effective decisions regarding the hardware specifications of the cluster nodes. We will explore in this article best practices to declare storage, CPU, and memory resources needs. We will also discuss how Kubernetes behaves if you don’t specify some of these dependencies.

Storage Dependency

Let’s explore the most common runtime requirement of an application: persistent storage. By default, any modifications made to the filesystem of a running container are lost when the container is restarted. Kubernetes provides two solutions to ensure that changes persist: emptyDir and Persistent Volumes.

Using Persistent Volumes, you can store data that does not get deleted even if the whole Pod was terminated or restarted. There are several methods by which you can provision a backend storage to the cluster. It depends on the environment where the cluster is hosted (on-prem or in the cloud and the cloud provider). In the following lab, we use the host’s disk as the persistent volume backend storage. Provisioning storage using Persistent Volumes involves two steps:

1. Creating the Persistent Volume: this is the disk on which Pods claim space. This step differs depending on the hosting environment.
2. Creating a Persistent Volume Claim: this is where you actually provision the storage for the Pod by claiming space on the Persistent Volume.

In the following lab, we create a Persistent Volume using the host’s local disk. Create a new YAML definition file, PV.yaml, and add the following lines:

apiVersion: v1
kind: PersistentVolume
metadata:
 name: hostpath-vol
spec:
 storageClassName: local
 capacity:
   storage: 1Gi
 accessModes:
   - ReadWriteOnce
 hostPath:
   path: "/tmp/data"

mkdir /tmp/data

$ kubectl apply -f PV.yaml
persistentvolume/hostpath-vol created

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: my-pvc
spec:
 storageClassName: local
 accessModes:
   - ReadWriteOnce
 resources:
   requests:
     storage: 100Mi
---
apiVersion: v1
kind: Pod
metadata:
 name: pvc-example
spec:
 containers:
 - image: alpine
   name: pvc-example
   command: ['sh', '-c', 'sleep 10000']
   volumeMounts:
     - mountPath: "/data"
       name: my-vol

 volumes:
   - name: my-vol
     persistentVolumeClaim:
      claimName: my-pvc

$ kubectl apply -f pvc_pod.yaml
persistentvolumeclaim/my-pvc created
pod/pvc-example created

kind: ConfigMap
apiVersion: v1
metadata:
 name: myconfigmap
data:
 # Configuration values can be set as key-value properties
 dbhost: db.example.com
 dbname: mydb
---
kind: Pod
apiVersion: v1
metadata:
 name: mypod
spec:
 containers:
   - name: mycontainer
     image: nginx
     envFrom:
       - configMapRef:
           name: myconfigmap

$ kubectl apply -f pod.yml
configmap/myconfigmap created
pod/mypod created
$ kubectl exec -it mypod -- bash
root@mypod:/# echo $dbhost
db.example.com
root@mypod:/# echo $dbname
mydb

However, this creates a dependency of its own: if the configMap was not available, the container might not work as expected. In our example, if this container an application that needs a constant database connection to work, then if it failed to obtain the database name and host, it may not work at all.

The same thing holds for Secrets, which must be available firsthand before any client containers can get spawned.

Resource Dependencies

So far we discussed the different runtime dependencies that affect which node will the Pod get scheduled (if at all) and the various prerequisites that must be availed for the Pod to function correctly. However, you must also take into consideration that capacity requirement of the containers.

Controllable and Uncontrollable Resources

When designing an application, we need to be aware of the type of resources that this application may consume. Generally, resources can be classified into two main categories:

• Shareable: those are the resources that can be shared among different consumers and, thus, limited when required. Examples of this are CPU and network bandwidth.
• Non-shareable: resources that cannot be shared by nature. For example, memory. If a container tries to use more memory than its allocation, it will get killed.

Declaring Pods Resource Requirements

The distinction between both resource types is crucial for a good design. Kubernetes allows you to declare the amount of CPU and memory the Pod requires to function. There are two parameters that you can use for this declaration:

• requests: this is the minimum amount of resources that the Pod needs. For example, you may already have the knowledge that the hosted application will fail to start if it does not have access to at least 512 MB of memory.
• limits: the limits define the maximum amount of resources that you need to supply for a given Pod.

Let’s have a quick example for a hypothetical application that needs at least 512 MB and 0.25% of a CPU core to run. The definition file for such a Pod may look like this:

kind: Pod
apiVersion: v1
metadata:
 name: mypod
spec:
 containers:
   - name: mycontainer
     image: myapp
     resources:
       requests:
         cpu: 250m
         memory: 512Mi
       limits:
         cpu: 500m
         memory: 750Mi

When the scheduler manages to deploy this Pod, it will search for a node that has at least 512 MB of memory free. If a suitable node was found, the Pod gets scheduled on it. Otherwise, the Pod will never get deployed. Notice that only the requests field is considered by the scheduler when determining where to deploy the Pod.

How Are the Resource Requests and Limits Calculated?

Memory is calculated in bytes, but you are allowed to use units like Mi and Gi to specify the requested amount. Notice that you should not specify a memory limit that is higher than the amount of memory on your nodes. If you did, the Pod would never get scheduled. Additionally, since memory is a non-shareable resource as we discussed, if a container tried to request more memory than the limit, it will get killed. Pods that are created through a higher controller like a ReplicaSet or a Deployment have their containers restarted automatically when they crash or get terminated. Hence, it is always recommended that you create Pods through a controller.

CPU is calculated through millicores. 1 core = 1000 millicores. So, if you expect your container needs at least half a core to operate, you set the request to 500m. However, since CPU belongs to shareable resources when the container requests more CPU than the limit, it will not get terminated. Rather, the Kubelet throttles the container, which may negatively affect its performance. It is advised here that you use liveness and readiness probes to ensure that your application’s latency does not affect your business requirements.

What Happens When You (not) Specify Requests and Limits?

Most of the Pod definitions examples ignore the requests and limits parameters. You are not strictly required to include them when designing your cluster. Adding or ignoring requests and limits affects the Quality of Service (QoS) that the Pod receives as follows:

Lowest priority pods: when you do not specify requests and limits, the Kubelet will deal with your Pod in a best-effort manner. The Pod, in this case, has the lowest priority. If the node runs out of non-shareable resources, best-effort Pods are the first to get killed.

Medium priority pods: if you define both parameters and set the requests to be less than the limit, then Kubernetes manages your Pod in the Burstable manner. When the node runs out of non-shareable resources, the Burstable Pods will get killed only when there are not more best-effort Pods running.

Highest priority pods: your Pod will be deemed as of the most top priority when you set the requests and the limits to equal values. It’s as if you’re saying, “I need this Pod to consume no less and no more than x memory and y CPU.” In this case, and in the event of the node running out of shareable resources, Kubernetes does not terminate those Pods until the best-effort, and the burstable Pods are terminated. Those are the highest priority Pods.

We can summarize how the Kubelet deals with Pod priority as follows:

Pod Priority and Preemption

Sometimes you may need to have more fine-grained control over which of your Pods get evicted first in the event of resource starvation. You can guarantee that a given Pod get evicted last if you set the request and limit to equal values. However, consider a scenario when you have two Pods, one hosting your core application and another hosting its database. You need those Pods to have the highest priority among other Pods that coexist with them. But you have an additional requirement: you want the application Pods to get evicted before the database ones do. Fortunately, Kubernetes has a feature that addresses this need: Pod Priority and preemption. Pod Priority and preemption is stable as of Kubernetes 1.14 or higher. The feature is enabled by default since Kubernetes version 1.11 (beta release). If your cluster version is less than 1.11, you will need to enable this feature explicitly.

So, back to our example scenario, we need two high priority Pods, yet one of them is more important than the other. We start by creating a PriorityClass then a Pod that uses this PriorityClass:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
 name: high-priority
value: 1000
---
apiVersion: v1
kind: Pod
metadata:
 name: mypod
spec:
 containers:
 - image: redis
   name: mycontainer
 priorityClassName: high-priority

The definition file creates two objects: the PriorityClass and a Pod. Let’s have a closer look at the PriorityClass object:

Line 1: the API version. As mentioned, PriorityClass is stable as of Kubernetes 1.14.

Line 2: the object type

Line 3 and 4: the metadata where we define the object name.

Line 5: we specify the value on which the priority is calculated relative to other Pods in the cluster. A higher value indicates a higher priority.

Next, we define the Pod that uses this PriorityClass by referring its name.

How Pods Get scheduled Given their PriorityClass Value?

When we have multiple Pods with different PriorityClass values, the admission controller starts by sorting Pods according to their priority. Highest priority Pods (those having the highest PriorityClass numbers) get scheduled first as long as no other constraints are preventing their scheduling.

Now, what happens if there are no nodes with available resources to schedule a high-priority pod? The scheduler will evict (preempt) lower priority Pods from the node to give enough room for the higher priority ones. The scheduler will continue lower-priority Pods until there is enough room to accommodate the more upper Pods.

This feature helps you when you design the cluster so that you ensure that the highest priority Pods (for example, the core application and database) are never evicted unless no other option is possible. At the same time, they also get scheduled first.

Things to Consider in Your Design When Using QoS and Pod Priority

You may be asking what happens when you use resources and limits (QoS) combined with the PriorityClass parameter. Do they overlap or override each other? In the following lines, we show you some of the essential things to note when influencing the schedule decisions:

• The Kubelet uses QoS to control and manage the node’s limited resources among the Pods. QoS eviction happens only when the node starts to run out of shareable resources (see the earlier discussion in this article). The Kubelet considers QoS before considering Preemption priorities.
• The scheduler considers the PriorityClass of the Pod before the QoS. It does not attempt to evict Pods unless higher-priority Pods need to be scheduled and the node does not have enough room for them.
• When the scheduler decides to preempt lower-priority pods, it attempts a clean shutdown and respects the grace period (thirty seconds by default). However, it does not honor PodDisruptionBudget, which may lead to disrupting the cluster quorum of several low priority Pods.

TL;DR

• On a single node environment, you can think of a container as a way of packaging and isolating applications. But when your environment spans across multiple nodes, you can also use containers as a means of efficient capacity planning.
• Determining the resources that your applications require at design time will save you a lot of time and effort afterwards. An application may have the following dependencies:
  • Volumes
  • Ports
  • Node resources like CPU and memory
• Part of the dependency fulfillment is handled by the cluster administrator (volumes, configMaps, Secrets,etc.) while the scheduler and the kubelet control the other part. You can still influence the decisions of both.
• You can design your Pods without defining the minimum and maximum resources they may need. However, doing so forces Kubernetes to use “best-effort” when scheduling your Pod, which may result in evicting more important Pods than others. You should be careful when setting the resource limits (requests and limits) because setting an incorrect value may lead to unnecessary eviction (when using non-shareable resources) or degraded performance (when using shareable resources).
• Setting the requests and limits parameters influences the kubelet decision as to which Pod to evict first in the event of resource starvation.
• Defining and setting the PriorityClass for a Pod influences the scheduler decision as to which Pods get scheduled first and which Pods get evicted if no room is available on the nodes.

Guest post by Ajmal Kohgadai, originally published on StackRox

By now most of us have heard about the role human error plays in causing data breaches. The Capital One breach from July is just the latest in a long line of security incidents that can trace their success back to a misconfigured infrastructure or security setting. As organizations accelerate their use of containers and Kubernetes and move their application development and deployment to cloud platforms, preventing avoidable misconfigurations in their environment becomes increasingly crucial.

Fortunately, most organizations understand that containers and Kubernetes are just like previous waves of infrastructure, where security starts with a securely configured infrastructure. In a recent survey of IT and security practitioners, respondents identified user-driven misconfigurations as their biggest concern for container security.

In this article, we will take a deep dive into key Kubernetes security configurations and recommended best practices you should follow.

It should be noted however that ensuring adherence to these best practices requires more than just knowing what they are. The level of success you have in consistently following these recommendations will also be determined by the degree to which you can automate the process of checking your environment for misconfigurations.

That’s because in a sprawling Kubernetes environment with several clusters spanning tens, hundreds, or even thousands of nodes, created by hundreds of different developers, manually checking the configurations is not feasible. And like all humans, developers can make mistakes – especially given that Kubernetes configuration options are complicated, security features are not enabled by default, and most of the community is learning how to effectively use components including Pod Security Policies and Security Context, Network Policies, RBAC, the API server, kubelet, and other Kubernetes controls.

As you and your teams come up to speed on all the details of Kubernetes security, follow these best practices to build a strong foundation:

1. Update Kubernetes to the latest version

If you haven’t already done so, update your Kubernetes deployments to the latest version (1.16), which includes several new and exciting features. Every new release is typically bundled with a host of different security features. Be sure to check out our blog post that highlights 7 reasons why you should upgrade Kubernetes to the latest version.

2. Use Pod Security Policies to prevent risky containers/Pods from being used

PodSecurityPolicy is a cluster-level resources available in Kubernetes (via kubectl) that is highly recommended. You must enable the PodSecurityPolicy admission controller to use it. Given the nature of admission controllers, you must authorize at least one policy – otherwise no pods will be allowed to be created in the cluster.

Pod Security Policies address several critical security use cases, including:

• Preventing containers from running with privileged flag – this type of container will have most of the capabilities available to the underlying host. This flag also overwrites any rules you set using CAP DROP or CAP ADD.
• Preventing sharing of host PID/IPC namespace, networking, and ports – this step ensures proper isolation between Docker containers and the underlying host
• Limiting use of volume types – writable hostPath directory volumes, for example, allow containers to write to the filesystem in a manner that allows them to traverse the host filesystem outside the pathPrefix, so readOnly: true must be used
• Putting limits on host filesystem use
• Enforcing read only for root file system via the ReadOnlyRootFilesystem
• Preventing privilege escalation to root privileges
• Rejecting containers with root privileges
• Restricting Linux capabilities to bare minimum in adherence with least privilege principles

Some of these attributes can also be controlled via securityContext. You can learn more about security context here. However, it’s generally recommended that you shouldn’t customize the pod-level security context but should instead use Pod Security Policies (see Recommendation #6 on how to apply these controls).

You can learn more about Pod Security Policies here.
You can learn more about admission controllers here and here.

3. Use Kubernetes namespaces to properly isolate your Kubernetes resources

Namespaces give you the ability to create logical partitions and enforce separation of your resources as well as limit the scope of user permissions. You can learn more about namespaces here.

4. Use Network Policies to segment and limit container and pod communication

Network Policies are used to determine how pods are allowed to communicate. Check out our blog post that takes a deep dive into building secure Kubernetes Network Policies.

5. Create policies to govern image provenance using the ImagePolicyWebhook

Prevent unapproved images from being used with the admission controller ImagePolicyWebhook to reject pods that use unapproved images including:

• Images that haven’t been scanned recently
• Images that use a base image that’s not whitelisted
• Images from insecure registries

You can learn more about ImagePolicyWebhook here.

6. Securely configure the Kubernetes API server

The Kubernetes API server handles all the REST API calls between external users and Kubernetes components.

Run the below command on your master node:

ps -ef | grep kube-apiserver

In the output, check to ensure that the:

• --anonymous-auth argument shows as false. This setting ensures that requests not rejected by other authentication methods are not treated as anonymous and therefore allowed against policy.
• --basic-auth-file argument isn’t there. Basic auth uses plaintext credentials, instead of the preferred tokens or certificates, for authentication.
• --insecure-allow-any-token argument isn’t there. This setting will ensure that only secure tokens that are authenticated are allowed.
• –kubelet-https argument either isn’t there or shows as true. This configuration ensures that connections between the API server and the kubelets are protected in transit via Transport Layer Security (TLS).
• --insecure-bind-address argument isn’t there. This configuration will prevent the API Server from binding to an insecure address, preventing non-authenticated and unencrypted access to your master node, which minimizes your risk of attackers potentially reading sensitive data in transit.
• --insecure-port argument shows as 0. This setting will prevent the API Server from serving on an insecure port, which would prevent unauthenticated and unencrypted access to the master node and minimize the risk of an attacker taking control of the cluster.
• --secure-port argument either doesn’t exist or shows up as an integer between 1 and 65535. The goal here is to make sure all your traffic is served over https with authentication and authorization.
• --profiling argument shows as false. Unless you’re experiencing bottlenecks or need to troubleshoot something that needs investigation, there’s no need for the profiler, and having it there unnecessarily opens you to exposure of system and program details.
• --repair-malformed-updates argument shows as false. This setting will ensure that intentionally malformed requests from clients are rejected by the API Server.
• --enable-admission-plugins argument is set with a value that doesn’t contain AlwaysAdmit. If you configure this setting to always admit, then it will admit requests even if they’re not explicitly allowed by the admissions control plugin, which would decrease the plugin’s effectiveness.
• --enable-admission-plugins argument is set with a value that contains AlwaysPullImages. This configuration ensures that users aren’t allowed to pull images from the node to any pod by simply knowing the name of the image. With this control enabled, images will always be pulled prior to starting a container, which will require valid credentials.
• --enable-admission-plugins argument is set with a value that contains SecurityContextDeny. This control ensures that you can’t customize pod-level security context in a way not outlined in the Pod Security Policy. See the Pod Security Policy section (#2) for additional information on security context.
• --disable-admission-plugins argument is set with a value that does not contain NamespaceLifecycle. You don’t want to disable this control, because it ensures that objects aren’t created in non-existent namespaces or in those namespaces set to be terminated.
• --audit-log-path argument is set to an appropriate path where you want your audit logs to be stored. It’s always a good security practice to enable auditing for any Kubernetes components, when available, including the Kubernetes API server.
• –-audit-log-maxage argument is set to 30 or whatever number of days you must store your audit log files to comply with internal and external data retention policies.
• --audit-log-maxbackup argument is set to 10 or any number that helps you meet your compliance requirements for retaining the number of old log files.
• --audit-log-maxsize argument is set to 100 or whatever number that helps you meet your compliance requirements. Note that number 100 represents 100 MB.
• --authorization-mode argument is there and is not set to AlwaysAllow. This setting ensures that only authorized requests are allowed by the API Server, especially in production clusters.
• --token-auth-file argument is not there. This argument, when present, uses static token-based authentication, which have several security flaws; use alternate authentication methods instead, such as certificates.
• --kubelet-certificate-authority argument is there. This setting helps prevent a man-in-the-middle attack when there’s a connection between the API Server and the kubelet.
• --kubelet-client-certificate and --kubelet-client-key arguments are there. This configuration ensures that the API Server authenticates itself to the kubelet’s HTTPS endpoints. (By default, the API Server doesn’t take this step.)
• --service-account-lookup argument is there and set to true. This setting helps prevent an instance where the API Server verifies only the validity of the authentication token without ensuring that the service account token included in the request is present in etcd.
• --enable-admission-plugins argument is set to a value that contains PodSecurityPolicy. See above section on Pod Security Policies (#2) for more details.
• --service-account-key-file argument is there and is set to a separate public/private key pair for signing service account tokens. If you don’t specify public/private key pair, it will use the private key from the TLS serving certificate, which would inhibit your ability to rotate the keys for service account tokens.
• --etcd-certfile and --etcd-keyfile arguments are there so that the API server identifies itself to the etcd server using client cert and key. Note that etcd stores objects that are likely sensitive in nature, so any client connections must use TLS encryption.
• --disable-admission-plugins argument is set and doesn’t contain ServiceAccount. This configuration will make sure that when a new pod is created, it will not use a default service account within the same namespace.
• --tls-cert-file and --tls-private-key-file arguments are there such that the API Server serves only HTTPS traffic via TLS.
• --client-ca-file argument exists to ensure that TLS and client cert authentication is configured for Kube cluster deployments.
• --etcd-cafile argument exists and it is set such that the API Server must verify itself to the etcd server via SSL Certificate Authority file.
• --tls-cipher-suites argument is set in a way that uses strong crypto ciphers.
• --authorization-mode argument is there with a value containing Node. This configuration limits which objects kubelets can read associated with their nodes.
• --enable-admission-plugins argument is set and contains the value NodeRestriction. This plugin ensures that a kubelet is allowed to modify only its own Node API object and those Pod API objects associated to its node.
• --encryption-provider-config argument is set to a EncryptionConfig file and this file should have all the needed resources. This setting ensures that all the REST API objects stored in the etcd key-value store are encrypted at rest.
• Make sure aescbc encryption provider is utilized for all desired resources as this provider of encryption is considered the strongest.
• --enable-admission-plugins argument contains the value EventRateLimit to set a limit on the number of events accepted by the API Server for performance optimization of the cluster.
• --feature-gates argument is not set with a value containing AdvancedAuditing=false. In other words, make sure advanced auditing is not disabled for auditing and investigation purposes.
• --request-timeout argument is either not set or set to an appropriate value (neither too short, nor too long). Default value is 60 seconds.
• --authorization-mode argument exists and is set to a value that includes RBAC. This setting ensures that Role-based access control (RBAC) is turned on. Beyond simply turning it on, you should follow several other recommendations for how to best use Kubernetes RBAC, including:
  • Avoid giving users cluster-admin role because it gives very broad powers over the environment and should be used very sparingly, if at all.
  • Audit your role aggregation rules to ensure you’re using them properly
  • Don’t grant duplicated permissions to subjects because it can make access revocation more difficult
  • Regularly remove unused Roles

7. Securely configure the kube-scheduler

As the default scheduler for Kubernetes, kube-scheduler selects the node that a newly created Pod should run on. You can learn more about kube-scheduler here.

Run the below command on your master node:

ps -ef | grep kube-scheduler

In the output, check to ensure that the:

• --profiling argument is set to false so that you have a reduced attack surface. While profiling can be useful when you have a performance bottleneck by identifying the bottleneck, it can also be exploited to reveal details about your system.
• --address argument is set to 127.0.0.1 so that the scheduler is not bound to a non-loopback insecure address, since the scheduler API service is available without authentication or encryption.

8. Securely configure the kube-controller-manager

Run the below command on your master node:

ps -ef | grep kube-controller-manager

In the output, check to ensure that the:

• --terminated-pod-gc-threshold argument is set to a value that ensures you have enough resources available and performance isn’t degraded.
• --profilingargument is set to false.
• --use-service-account-credentials argument is set to true. When combined with RBAC, this setting ensures that control loops run with minimum permissions required to adherence to least privilege design principles.
• --service-account-private-key-file argument is set such that a separate public/private key pair is used for signing service account tokens.
• --root-ca-file argument exists and is set to a cert file containing the root cert for the API Server’s serving cert, which will allow pods to verify the API Server’s serving cert before making a connection.
• RotateKubeletServerCertificate argument is there and set as true, and applies only when kubelets get their certs from the API Server.
• --address argument is set to 127.0.0.1, so that the controller manager service is not bound to non-loopback insecure addresses.

Secure the configuration files on the master node

Secure the API server pod specification file permissions.

Run the following command on the master node (specifying your file location on your system):

stat -c %a /etc/kubernetes/manifests/kube-apiserver.yaml

In the output, check to ensure that permissions are 644 or more restrictive to maintain the integrity of the file.

Secure the API Server pod specification file ownership.

Run the following command on the master node (specifying your file location on your system):

stat -c %U:%G /etc/kubernetes/manifests/kube-apiserver.yaml

In the output, check to ensure that ownership is set as root:root to maintain the integrity of the file.

Secure the controller manager pod specification file permissions.

Run the following command on the master node (specifying your file location on your system):

stat -c %a /etc/kubernetes/manifests/kube-controller-manager.yaml

In the output, check to ensure that permissions are 644 or more restrictive to maintain the integrity of the file.

Secure the controller manager pod specification file ownership.

Run the following command on the master node (specifying your file location on your system):

stat -c %U:%G /etc/kubernetes/manifests/kube-controller-manager.yaml

In the output, check to ensure that ownership is set as root:root to maintain the integrity of the file.

Secure the scheduler pod specification file permissions.

Run the following command on the master node (specifying your file location on your system):

stat -c %a /etc/kubernetes/manifests/kube-scheduler.yaml

In the output, check to ensure that permissions are 644 or more restrictive to maintain the integrity of the file.

Secure the scheduler pod specification file ownership.

Run the following command on the master node (specifying your file location on your system):

stat -c %U:%G /etc/kubernetes/manifests/kube-scheduler.yaml

In the output, check to ensure that ownership is set as root:root to maintain the integrity of the file.

Secure the etcd pod specification file permissions.

Run the following command on the master node (specifying your file location on your system):

stat -c %a /etc/kubernetes/manifests/etcd.yaml

In the output, check to ensure that permissions are 644 or more restrictive to maintain the integrity of the file. As a reminder on a topic already discussed, etcd is a key-value store, and protecting it is of the utmost importance, since it contains your REST API objects.

Secure the etcd pod specification file ownership.

Run the following command on the master node (specifying your file location on your system):

stat -c %U:%G /etc/kubernetes/manifests/etcd.yaml

In the output, check to ensure that ownership is set as root:root to maintain the integrity of the file.

Secure the Container Network Interface file permissions.

Run the following command on the master node (specifying your file location on your system):

stat -c %a <path/to/cni/files>

In the output, check to ensure that permissions are 644 or more restrictive to maintain the integrity of the file.

Secure the Container Network Interface file ownership.

Run the following command on the master node (specifying your file location on your system):

stat -c %U:%G <path/to/cni/files>

In the output, check to ensure that ownership is set as root:root to maintain the integrity of the file.

Secure the etcd data directory permissions.

First run the following command to the get etcd data directory:

ps -ef | grep etcd

Now run the following command based on the etcd data directory you found from the previous command:

stat -c %a /var/lib/etcd

In the output, check to ensure that permissions are 700 or more restrictive to ensure your etcd data directory is protected against unauthorized reads/writes.

Secure the etcd data directory ownership.

First run the following command to the get etcd data directory:

ps -ef | grep etcd

Now run the following command based on the etcd data directory you found from the previous command:

stat -c %U:%G /var/lib/etcd

In the output, check to ensure that ownership is etcd:etcd to ensure your etcd data directory is protected against unauthorized reads/writes.

Secure the admins.conf file permissions.

Run the following command on the master node (specifying your file location on your system):

stat -c %a /etc/kubernetes/admin.conf

In the output, check to ensure that permissions are 644 or more restrictive to maintain the integrity of the file.

Secure the admins.conf file ownership.

Run the following command on the master node (specifying your file location on your system):

stat -c %U:%G /etc/kubernetes/admin.conf

In the output, check to ensure that ownership is set as root:root to maintain the integrity of the file.

Secure the scheduler.conf file permissions.

Run the following command on the master node (specifying your file location on your system):

stat -c %a /etc/kubernetes/scheduler.conf

In the output, check to ensure that permissions are 644 or more restrictive to maintain the integrity of the file.

Secure the scheduler.conf file ownership.

Run the following command on the master node (specifying your file location on your system):

stat -c %U:%G /etc/kubernetes/scheduler.conf

In the output, check to ensure that ownership is set as root:root to maintain the integrity of the file.

Secure the controller-manager.conf file permissions.

Run the following command on the master node (specifying your file location on your system):

stat -c %a /etc/kubernetes/controller-manager.conf

In the output, check to ensure that permissions are 644 or more restrictive to maintain the integrity of the file.

Secure the controller-manager.conf file ownership.

Run the following command on the master node (specifying your file location on your system):

stat -c %U:%G /etc/kubernetes/controller-manager.conf

In the output, check to ensure that ownership is set as root:root to maintain the integrity of the file.

Secure the Kubernetes PKI directory and file ownership.

Run the following command on the master node (specifying your file location on your system):

ls -laR /etc/kubernetes/pki/

In the output, check to ensure that ownership is set as root:root to maintain the integrity of the file.

Secure the Kubernetes PKI directory and file permissions.

Run the following command on the master node (specifying your file location on your system):

ls -laR /etc/kubernetes/pki/*.crt

In the output, check to ensure that permissions are 644 or more restrictive to maintain the integrity of the file.

Secure the Kubernetes PKI key file permissions.

Run the following command on the master node (specifying your file location on your system):

ls -laR /etc/kubernetes/pki/*.key

In the output, check to ensure that permissions are 600 to maintain the integrity of the file.

10. Securely configure etcd

As mentioned in previous sections, etcd (a CNCF project) is a key-value store (a CNCF project) used by distributed systems such as Kubernetes for data access. etcd is considered the source of truth for Kubernetes, and you can read data from and write into etcd as needed. Securely configuring etcd and communications to its servers are of utmost criticality.

Run the following command on the etcd server node:

ps -ef | grep etcd

In the output, check to ensure that the:

• --cert-file and the --key-file arguments are set as needed to ensure client connections are served only over TLS (in transit encryption).
• --client-cert-auth argument shows as true to ensure all access attempts from clients include a valid client cert.
• --auto-tls argument is there and is not true, or isn’t there at all, which will prohibit clients from using self-signed certs for TLS.
• If you’re using a etcd cluster (instead of a single etcd server), check to see that --peer-cert-file and --peer-key-file arguments are appropriately set to ensure etcd peer connections is encrypted within the etcd cluster. In addition, check that --peer-client-cert-auth argument is set to true, as this setting would ensure that only authenticated etcd peers can access the etcd cluster. Lastly verify that if --peer-auto-tls argument is there, it is not set to true.
• As a best practice, don’t use the same certificate authority for etcd as you do for Kubernetes. You can ensure this separation by verifying that the file referenced by the --client-ca-file for API Server is different from the --trusted-ca-file used by etcd.

11. Securely configure the Kubelet

The kubelet is the main “node agent” running on each node. Misconfiguring kubelet can expose you to a host of security risks, as this Medium article last year outlines. You can either use arguments on the running kubelet executable or a kubelet config file to set the configuration of your kubelet.

To find the kubelet config file, run the following command:

ps -ef | grep kubelet | grep config

Look for --config argument, which will give you the location of the kubelet config file.

Then run the following command on each node:

ps -ef | grep kubelet

In the output, make sure that the:

• --anonymous-auth argument is false. In the kubelet article previously referenced, one of the misconfigurations exploited was one where anonymous (and unauthenticated) requests were allowed to be served by the kubelet server.
• --authorization-mode argument shows as AlwaysAllow if it’s there. If it is not there, make sure there’s a kubelet config file specified by --config and that file has set authorization: mode to something besides AlwaysAllow.
• --client-ca-file argument is there and set to the location of the client certificate authority file. If it’s not there, make sure there’s a kubelet config file specified by --config and that file has set authentication: x509: clientCAFile to the location of the client certificate authority file.
• --read-only-port argument is there and set to 0. If it’s not there, make sure there’s a kubelet config file specified by --config, and readOnlyPort is set to 0 if it’s there.
• --protect-kernel-defaults shows as true. If it’s not there, make sure there’s a kubelet config file specified by --config, and that file has set protectKernelDefaults as true.
• --hostname-override argument is not there, to ensure that the TLS setup between the kubelet and the API Server doesn’t break.
• --event-qps argument is there and set to 0. If it’s not there, make sure there’s a kubelet config file specified by --config and eventRecordQPS shows as 0.
• --tls-cert-file and --tls-private-key-file arguments are set appropriately or the kubelet config specified by --config contains appropriate settings for tlsCertFile and tlsPrivateKeyFile. This configuration ensures that all connections happen over TLS on the kubelets.
• RotateKubeletServerCertificate and --rotate-certificates is set to true if your kubelets get their certs from the API Server, and make sure your kubelet uses only strong crypto ciphers.

12. Secure the worker node configuration files

Secure the kubelet service file permissions.

Run the following command on each worker node (specifying your file location on your system):

stat -c %a /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

In the output, check to ensure that permissions are 644 or more restrictive to maintain the integrity of the file.

Secure the kubelet.conf file permissions.

Run the following command on each worker node (specifying your file location on your system):

stat -c %a /etc/kubernetes/kubelet.conf

In the output, check to ensure that permissions are 644 or more restrictive to maintain the integrity of the file.

Secure the kubelet.conf file ownership.

Run the following command on each worker node (specifying your file location on your system):

stat -c %U:%G /etc/kubernetes/kubelet.conf

In the output, check to ensure that ownership is set as root:root to maintain the integrity of the file.

Secure the kublete service file ownership.

Run the following command on each worker node (specifying your file location on your system):

stat -c %U:%G /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

In the output, check to ensure that ownership is set as root:root to maintain the integrity of the file.

Secure the proxy kubeconfig file permissions.

Run the following command to first find the kubeconfig file being used:

ps -ef | grep kube-proxy

Get the kube-proxy file location (if it’s running) from --kubeconfig, then run the following command on each worker node (specifying your file location on your system).

stat -c %a <proxy kubeconfig file>

In the output, check to make sure permissions are 644 or more restrictive to maintain the integrity of the file.

Secure the proxy kubeconfig file ownership.

Run the following command first to find the kubeconfig file being used:

ps -ef | grep kube-proxy

Get the kube-proxy file location (if it’s running) from --kubeconfig, then run the following command on each worker node (specifying your file location on your system):

stat -c %U:%G <proxy kubeconfig file>

In the output, check to make sure ownership is set as root:root to maintain the integrity of the file.

Secure the certificate authorities file permissions.

Run the following command first:

ps -ef | grep kubelet

Look for the file name that’s identified by --client-ca-file argument. Then run the following command, specifying the previous file name:

stat -c %a <filename>

In the output, check to make sure permissions are 644 or more restrictive to maintain the integrity of the file.

Secure the client certificate authorities file ownership.

Run the following command first:

ps -ef | grep kubelet

Look for the file name that’s identified by –-client-ca-file argument. Then run the following command, specifying the previous file name:

stat -c %U:%G <filename>

In the output, check to make sure ownership is set asroot:root to maintain the integrity of the file.

Secure the kubelet configuration file permissions.

First locate the kubelet config file with following command:

ps -ef | grep kubelet | grep config

In the output, you may see the location of the config file if it exists. It would look something like /var/lib/kubelet/configuration.yaml.

Using the location of the file (we’ll use the file location from this previous example), run the following command to identify the file’s permissions:

stat -c %a /var/lib/kubelet/configuration.yaml

In the output, check to make sure permissions are set to 644 or more restrictive to ensure the integrity of the file.

Secure the kubelet configuration file ownership.

Run the following command:

ps -ef | grep kubelet | grep config

In the output, you may see the location of the config file if it exists – it would look something like /var/lib/kubelet/configuration.yaml.

Using the location of the file (we’ll use the file location from this previous example), run the following command to identify the file’s permissions:

stat -c %U:%G /var/lib/kubelet/configuration.yaml

In the output, check to make sure ownership is set to root:root to maintain the integrity of the file.

This cloud-native stack offers compelling capabilities for building the most secure applications we’ve ever created – we just need to make sure we’ve got all the knobs and dials set correctly. Leverage these configurations, code examples, and detailed recommendations to avoid the security risks associated with the most common Kubernetes misconfigurations.

We are excited to announce that we’re now accepting applications to run Kubernetes Community Days around the world. 

Kubernetes Community Days are community-organized events that gather adopters and technologists from open source and cloud native communities to learn, collaborate, and network to further the adoption and improvement of Kubernetes. 

The Cloud Native Computing Foundation supports Kubernetes Community Days by providing guidance and tools, covering all the aspects of holding a successful event. These events are decentralized and focused on community engagement. We hope they will be a lot of fun, and prove to be a great way to meet new people while also building community. Local event organizers handle their own sponsorships, registration, and all other logistics. Each event brings its own local flair, culture, diversity, and authenticity (and its own logo and t-shirts, of course!)

We would like to give a hearty thank you to DevOpsDays and OpenStack Days for their inspiration and, in particular, for providing the basis of this program. We liberally borrowed from their thoughts, their experience, and their documentation.

Learn more about hosting your own Kubernetes Community Day.  

Today we are very excited to release our Project Journey Report for Envoy. This is the second reports we have issued  for CNCF graduated projects (the first was Kubernetes).

Envoy is a widely-adopted, open source network proxy developed by engineers at Lyft and released on September 14, 2016. It is frequently used in conjunction with deployments of Kubernetes and other cloud native technologies, however is used in many non-cloud environments also.

CNCF’s efforts to nurture the growth of Envoy span a wide range of activities from organizing and running the rapidly growing EnvoyCon to creating webinars and recording case studies to make Envoy more accessible and to foster and nurture the Envoy community. These activities are funded by CNCF’s membership dues and revenues from sponsorship and registration at our conferences.

This is one of a series of reports that help explain our nurturing efforts and some of the positive trends emerging around CNCF hosted projects. This report attempts to objectively assess the state of the Envoy project and how the CNCF has impacted the progress and growth of Envoy. Note – we recognize that it’s not feasible to conclusively sort out correlation and causation but this report tries to document correlations. For the report, we pulled data from multiple sources, particularly CNCF’s own DevStats tool, which provides detailed development statistics for all CNCF projects.

Some of the highlights of the report include:

• Development Velocity – Envoy continues to show strong growth across all four of the key development velocity vectors – code commits, pull requests, issues filed, and authors. Envoy has enjoyed over a 600% increase in the number of project contributors since Envoy joined CNCF.

Cumulative growth of Envoy contributors over time

• Code Diversity – Envoy has actually grown out of the end user community and that community continues to contribute a large percentage of Envoy’s. Additionally Envoy started primarily as a Lyft/Google project but today over four dozen companies are regularly contributing to the project, spanning the world’s largest cloud computing companies down to small startups and individual contributors. 

Percentage breakdown of contributions by company since Envoy project launch

• Documentation Expansion – Continuous additions to and improvements of project documentation are essential for the growth of any open source project. Since joining CNCF, the number of authors and companies committing documentation to Envoy has grown by more than 300% and 200%, respectively. 

Since joining CNCF in 2017, Envoy has recorded:

• >1.7K contributors
• >10.3K code commits
• >5.7 K pull requests 
• >51k contributions
• 176 contributing companies

We’re thrilled that Envoy has come so far in three years and that CNCF has been able to positively impact that growth in the past two years of our collaboration with the Envoy community. Even better, we strongly believe Envoy’s growth has only just begun. 

This report is part of CNCF’s commitment to fostering and sustaining an ecosystem of open source, vendor-neutral projects. Please read and enjoy the report, share your feedback with us – and stay tuned for more project journey reports for other projects.

Cars that update like smartphones, adjusting features based on the driver’s preferences? The future is now for Japan’s DENSO Corporation, one of the biggest automotive components suppliers in the world. 

With the advent of connected cars, DENSO established a Digital Innovation Department to expand its business beyond the critical layer of the engine, braking systems, and other automotive parts into the non-critical analytics and entertainment layer. Comparing connected cars to smartphones, R&D Product Manager Seiichi Koizumi says DENSO wants the ability to quickly and easily develop and install apps for the “blank slate” of the car, and iterate them based on the driver’s preferences. Thus “we need a flexible application platform,” he says. 

But working on vehicle edge and vehicle cloud products meant there were several technical challenges: “the amount of computing resources, the occasional lack of mobile signal, and an enormous number of distributed vehicles,” says Koizumi. “We are tackling these challenges to create an integrated vehicle edge/cloud platform.”

With a vehicle edge computer, a private Kubernetes cloud, and managed Kubernetes on GKE, EKS, and AKS, DENSO now has a 2-month development cycle for non-critical apps, compared to 2-3 years for traditional, critical-layer development. The company releases 10 new applications a year. And cloud native technologies have enabled DENSO to deliver these applications via its new dash cam, which has a secure connection that collects data to the cloud. 

The Digital Innovation Department is known as “Noah’s Ark,” and it has grown from 2 members to 70—with plans to more than double in the next year. The way they operate is completely different from the traditional Japanese automotive culture. But just as the company embraced change brought by hybrid cars in the past decade, Koizumi says, they’re doing it again now, as technology companies have moved into the connected car space. “Another disruptive innovation is coming,” he says, “so to survive in this situation, we need to change our culture.”

For more about DENSO’s cloud native journey, read the full case study.

Guest post from Kiran Mova and Chuck Piercey, MayaData

BigData, AI/ML and modern analytics permeate the business world and have become a critical element of enterprise strategies to serve customers better, innovate faster and stay ahead of the competition. Data is core to all of this. In this blog, we focus on how the Kubernetes and related container native storage technologies are enabling the data engineers (aka DataOps teams) to build scalable, agile data infrastructure that achieves these goals. 

Being a Data-Driven Enterprise is Strategic

Enterprises are increasing their spending to enable data driven decisions and foster a data driven culture. A recent survey of enterprise executives’ investments showed the importance of data-driven analytics to the C-suite.

One data point to highlight is the fact that people and process pose greater challenges to adoption than technology and tools.  

What is DataOps

Kubernetes has changed the landscape of application development. With emerging data operators in Kubernetes-managed data infrastructures, we have entered a new world of self-managed data infrastructures called DataOps. Inspired by DevOps, DataOps is a way to enable collaboration between distributed autonomous teams’ data analysts, data scientists and data engineers with shared KPIs. 

“DataOps stems from the DevOps movement in the software engineering world which bridges the traditional gap between development, QA, and operations so the technical teams can deliver high-quality output at an ever-faster pace. Similarly, DataOps brings together data stakeholders, such as data architects, data engineers, data scientists, data analysts, application developers, and IT operations….DataOps applies rigor to developing, testing, and deploying code that manages data flows and creates analytic solutions.” – Wayne W. Eckerson DataOps white paper

A key aspect of Kubernetes success is that DevOps can drive everything through versioned-managed Intent/YAML files (aka GitOps) to manage infrastructures the same way IT manages code reproducibly and scalably. The same approach can be applied to DataOps.

Siloed Teams & Processes Fail at DataOps

Data pipelines produced by organizational silos built around specialized teams (still in use by many businesses today) will suffer from having the delivery responsibility split across those functional groups. The organizational structure constrains the development process to something that can bridge the gaps between tools and this approach will ineluctably be prone to failures caused by the bureaucracy itself.

Integrated Teams & Processes Succeed at DataOps

A recent article on Distributed Data Mesh proposes many organizational and technical changes to how Data Engineering / Science teams can become more effective and agile, like the success seen by product development teams using DevOps/SRE culture. A shift from a tool focus to a CI/CD process focus is core to the paradigm. Both process and development execution necessarily co-evolve and, unsurprisingly, such teams often are themselves distributed, much like the architecture they build on. In his paper, Zhamak Dehghani proposes an approach for managing data as a product that parallels the DevOps techniques applied to commercial software products and teams.

The core of his approach is to redefine the team and its responsibilities by shifting the architectural milieu from a focus on technology and tools (like Data Engineers, ML Engineers, Analytics Engineers) to a more interdisciplinary concept structured around treating Data itself as a Product. In this structure, each Data Product has an independent team that can innovate, select tools, and implement while exposing data results using a standard API contract. 

These teams consist of a variety of skill sets: data engineers, data scientists, data analysts, ML engineers, decision makers, data collection specialists, data product managers, and reliability engineers (while the roles for successful data product teams are clear, individuals can simultaneously fulfill multiple roles). Critically, all team members interact with a shared set of DataOps processes.

Declarative Data Infrastructure

DataOps relies on a data infrastructure that can abstract away platform-specific features and allow product teams to focus on the data they own while leveraging shared resources. The key enabling technology for DataOps is Declarative Data Infrastructure (DDI). DDI refers to both the data and the storage infrastructure running on Kubernetes and is the technology stack that converts compute, network, and storage into a scalable, resilient and self-managed global resource that each autonomous team can use without having to wait on approvals from central storage administrators. Kubernetes and related technologies have emerged as a standard that enables the DDI technology stack.

For example, the data infrastructure in the data mesh example above is comprised of three layers:

• A Data Pipeline – like the Airflow framework
• A Data Access Layer – like Apache Kafka or Postgres 
• A Data Storage Layer – like OpenEBS.  

In the past couple of years, we have seen quite a few technologies under the Data Pipeline and Data Access Layer move towards Kubernetes. For instance, it isn’t uncommon for data engineering teams to move away from Hadoop-based systems to something like Pachyderm, moving their data pipelines into Kubernetes with Airflow to reduce the cost of infrastructure and create reproducible, resilient and extensible data pipelines.

Kubernetes projects like Airflow have reached maturity on the data pipelines’ implementation and orchestration over the past two years and are being adopted at companies like Lyft, Airbnb, Bloomberg.

Correspondingly, data pipeline’s adoption has triggered and enabled a new breed of products and tools for the Data Access and Storage Layers it feeds. 

An excellent example that demonstrates the move of Data Access Layer into Kubernetes is the Yolean prebuilt Kafka cluster on Kubernetes that delivers a production-quality, customizable Kafka-as-a-service Kubernetes cluster. Recently, Intuit used this template to rehost all their financial applications onto Kubernetes. 

Two components have made the data access layer easier. The first is a shim layer that provides declarative YAMLs for instantiating the Data Access Layer (like Kafka) for either an on-prem version or a managed service version. This approach to running the Data Access Layer helps users migrate from current implementations into Kubernetes. It suffers by locking users into specific implementations. An alternate approach is to have the Operators build the data access layer so that it can run on any storage layer, thereby avoiding cloud vendor lock-in. 

The data storage layer is seeing a similar shift, responding in part to the rise of new, inherently distributed workloads and the emergence of a new set of users as well. There are several Kubernetes native storage solutions that are built using the same declarative philosophy and managed by Kubernetes itself, that we refer as the Declarative Data Plane. 

Declarative Data Plane

The data plane is composed of two parts:

1. An Access Layer which primarily concerns with the access to the storage, and
2. Data Services like replication, snapshot, migration, compliance and so forth. 

CSI, a standard Declarative Storage Access Layer

Kubernetes and the CNCF vendor and end user community have been able to achieve a vendor neutral standard in the form of CSI to enable any storage vendors to provide storage to the Kubernetes  workloads. The workloads can be running on any type of container runtime – docker or hypervisors. In part, the success of some of the self-managed Data Access Layer products can be attributed to CSI as a standard and constructs like Storage Classes, PVCs and Customer Resources and Operators. 

However, CSI leaves many core data infrastructure implementation details to vendors and service, such as:

• Data Locality and high availability
• Compliance for GDPR or HIPPA
• Multi-Cloud and Hybrid-Cloud deployments
• Analytics and Visibility into usage for various teams

Thankfully, the extensibility of Kubernetes via the customer resources and operators is enabling a new breed of storage technologies that are Kubernetes native and sometimes called Container Attached Storage (CAS). CAS declaratively manages data services down to the storage device. 

Declarative (Composable) Data Services

Depending on the application requirements, a data plane can be constructed using one of the many options available. Kubernetes and containers have helped redefine the way Storage or the Data Plane is implemented.

For example, for distributed applications like Kafka, that has inbuilt replication, rebuilding, capabilities – a Local PV is just what is needed for the Data Plane. To use Local PVs in production, that typically involves provisioning, monitoring and managing backup/migration –  data infrastructure engineers just need to enable deploying of the Kubernete native tools like – Operators (that can perform static or dynamic provisioning), Prometheus, Grafana, Jaeger for observability,  Velero – for backup/restore.

For another set of applications, there may be a need to deploy a Data Plane that can perform replication – within a cluster, across clusters, across zones/clouds, and snapshots and cloning. By making use of the Customer Resources and Operators, Kubernetes native projects like Rancher, Longhorn and OpenEBS have emerged to combat cloud vendor lock-in for the Kubernetes ecosystem, not only providing a unified experience and tools for managing storage resources on-prem or cloud, but also leveraging and optimizing investments that Enterprises have already made in Legacy storage. 

 The Next Gen: Declarative Data Plane

With a unified experience and declarative interface to manage the storage/data services, data engineers can interact with Data Infrastructure in a standard way. Building on the CSI foundation, projects like OpenEBS, Velero, standards like KubeMove, SODA Foundation (aka Open Data Autonomy/OpenSDS) are focusing on implementing Easy-To-Use Kubernetes Storage Services for on-prem and cloud and are pushing forward standardization of Declarative Data Plane  (aka DDP).

The DDP delivers several architecturally important elements for the next generation of distributed applications’ DataOps: 

• Enabling autonomous teams to manage their own storage
• Scalable polyglot big data storage
• Encryption for data at rest and in motion
• Compute and data locality
• Compliance for GDPR or HIPPA
• Multi-Cloud and Hybrid-Cloud deployments
• Backup and Migration 
• Analytics and Visibility into usage for various teams 

The DDI project is backed by Infrastructure Teams in large Enterprises that have already adopted Kubernetes and are using Kubernetes and projects like OpenEBS to deliver: 

• Etcd As a Service
• ElasticSearch As a Service
• PostgreSQL As a Service
• ML pipelines of many types (one promising one is MELTANO from GitHub)
• Kafka as a service

These implementations show that enterprise customers are taking full advantage of the capabilities delivered by a Declarative Data Infrastructure. Such enterprises are leveraging the significant architectural enhancements DDI at the data layer provides to deliver faster, better and competitively-differentiating enterprise analytics. DDI helps with optimal use of the

Infrastructure / Cost Optimization.

Declarative Data Infrastructures are Here to Stay

The declarative (GitOps) approach to managing data infrastructure is here to stay. We recently heard a top executive at a large enterprise say that unless the technology stack is declarative, it is not operable. 

We are pleased to announce the schedule for the inaugural ServiceMeshCon, a KubeCon + CloudNativeCon co-located event. Hosted by CNCF, the conference will take place on November 18th on Day Zero of KubeCon + CloudNativeCon San Diego. 

ServiceMeshCon is a vendor neutral conference on service mesh technologies. The line-up will feature maintainers across various service mesh projects and showcase lessons learned from running this technology in production. 

Sessions will range from introductory-level to advanced, featuring speakers from companies that are both innovating and using service mesh technologies. Talks include:

• An Intro to Network Service Mesh (NSM) and its relationship to Service Mesh – John Joyce & Tim Swanson, Cisco
• Control Plane for Large Mesh in a Heterogeneous Environment – Fuyuan Bie & Zhimeng Shi, Pinterest
• How Google manages sidecars for millions of containers without breaking anything (much) – Sven Mawson, Google
• Service Mesh Interface: Developer friendly APIs for Service Mesh – Michelle Noorali, Microsoft
• There’s A Bug in My Service Mesh! What Do You Do When the Tool You Rely On is the Cause? – Ana Calin, Paybase

Tickets are available for $199 and pre-registration is required to attend ServiceMeshCon. Attendees can add it on during registration for KubeCon + CloudNativeCon San Diego.