All Posts By

Kaitlyn Barnard

This Week in Kubernetes: February 5th

By | Blog

Each week, the Kubernetes community shares an enormous amount of interesting and informative content including articles, blog posts, tutorials, videos, and much more. We’re highlighting just a few of our favorites from the week before. This week we’re talking machine learning, scalability, service mesh, and contributing to Kubernetes.

The Full-Time Job of Keeping up With Kubernetes, Gravitational

If you’ve ever wonder what goes into maintaining an open-source project with as much velocity as Kubernetes, this article is a great place to start. Abraham Ingersoll from Gravitational dives into the inner workings of SIGs and working groups, the Kubernetes release cycle, feature development process, and community support.

When to use Serverless? When to use Kubernetes?, Heidloff.Net

As the cloud native landscapes evolve, more and more questions come up as to when to use Kubernetes and when to use Serverless to build an application. In this article, Niklas Heidloff from IBM outlines the pros and cons of each option and how to decide what’s best for your use case.

Scheduling in Kubernetes, Alexandrutopliceanu.Ro

Scheduling in Kubernetes helps ensure that pods are only placed on nodes that have sufficient free resources. In this post, Alexandru Topliceanu from walks you through the implementation of the default scheduler in Kubernetes. Dive into the genericScheduler, volumes, algorithm, predicates, custom scheduler, and more to learn how to support long-running processes.

Kubernetes Autoscaling Based on Custom Metrics Without Using a Host Port, Medium

Autoscaling is one of the most useful features in Kubernetes, but autoscaling based on custom metrics can be complicated to set up since it’s still an alpha feature. In this Medium post, Marko Lukša from Red Hat shows you how to set up horizontal pod autoscaling based on application-provided custom metrics on minikube.

Stay tuned for more exciting content from the Kubernetes community next week, and join the KubeWeekly mailing list for the latest updates delivered directly to your inbox.

Is there a piece of content that you think we should highlight? Tweet at us! We’d love to hear what you’re reading, watching, or listening to.

This Week in Kubernetes: January 29th

By | Blog

Each week, the Kubernetes community shares an enormous amount of interesting and informative content including articles, blog posts, tutorials, videos, and much more. We’re highlighting just a few of our favorites from the week before. This week we’re talking machine learning, scalability, service mesh, and contributing to Kubernetes.

Q&A on Machine Learning and Kubernetes with David Aronchick of Google

Machine learning has been gaining a lot of attention, especially at KubeCon in Austin, TX this past December. Rags Srinivas of InfoQ sat down with David Aronchick, product manager at Google and contributor to Kubeflow, to discuss Kubernetes and machine learning. Take a look at how machine learning is changing businesses today, and how Kubernetes offers a common platform for deploying and running ML platforms at scale.

Scaling Kubernetes to 2,500 Nodes

OpenAI, has been running Kubernetes for deep learning research for the past two years. Kubernetes allows for fast iteration cycle, reasonable scalability, and lack of boilerplate making experimentation at OpenAI quick and easy. Learn how OpenAI scaled its Kubernetes clusters to more than 2,500 nodes on both the cloud and physical hardware, while remaining incident free for 90 days.

Kubernetes Service Mesh

Hearing a lot about service mesh recently and wondering what your options are? On his personal blog, Alen Komljen of Semantext Group explains why service mesh is a critical component of cloud-native. From load balancing and service discovery to service monitoring and tracing, this is a great introductory post to check out if you’re interested in service mesh. Alen also dives into Conduit and Istio to showcase how these tools work.

Riding the Unicorn: A Newbie Contributor’s Guide to Kubernetes

Arun Gupta wrote this great guide on the AWS Open Source Blog for anyone who is interested in contributing to Kubernetes. The Kubernetes community is growing fast, with increasing opportunities for new contributors. Check out this guide for some ideas of where to start, from joining the conversation on Slack and community meetings to getting involved in a Special Interest Group (SIG) and much more.

Stay tuned for more exciting content from the Kubernetes community next week, and join the KubeWeekly mailing list for the latest updates delivered directly to your inbox.

Is there a piece of content that you think we should highlight? Tweet at us! We’d love to hear what you’re reading, watching, or listening to.

Enforcing Network Policies in Kubernetes

By | Blog

Editor’s note: this post is part of a series of in-depth articles on what’s new in Kubernetes 1.8. Today’s post comes from Ahmet Alp Balkan, Software Engineer, Google.

Kubernetes now offers functionality to enforce rules about which pods can communicate with each other using network policies. This feature is has become stable Kubernetes 1.7 and is ready to use with supported networking plugins. The Kubernetes 1.8 release has added better capabilities to this feature.

Network policy: What does it mean?

In a Kubernetes cluster configured with default settings, all pods can discover and communicate with each other without any restrictions. The new Kubernetes object type NetworkPolicy lets you allow and block traffic to pods.

If you’re running multiple applications in a Kubernetes cluster or sharing a cluster among multiple teams, it’s a security best practice to create firewalls that permit pods to talk to each other while blocking other network traffic. Networking policy corresponds to the Security Groups concepts in the Virtual Machines world.

How do I add Network Policy to my cluster?

Networking Policies are implemented by networking plugins. These plugins typically install an overlay network in your cluster to enforce the Network Policies configured. A number of networking plugins, including Calico, Romana and Weave Net, support using Network Policies.

Google Container Engine (GKE) also provides beta support for Network Policies using the Calico networking plugin when you create clusters with the following command:

gcloud beta container clusters create –enable-network-policy

How do I configure a Network Policy?

Once you install a networking plugin that implements Network Policies, you need to create a Kubernetes resource of type NetworkPolicy. This object describes two set of label-based pod selector fields, matching:

  1. a set of pods the network policy applies to (required)
  2. a set of pods allowed access to each other (optional). If you omit this field, it matches to no pods; therefore, no pods are allowed. If you specify an empty pod selector, it matches to all pods; therefore, all pods are allowed.

Example: restricting traffic to a pod

The following example of a network policy blocks all in-cluster traffic to a set of web server pods, except the pods allowed by the policy configuration.

To achieve this setup, create a NetworkPolicy with the following manifest:

kind: NetworkPolicy
 name: access-nginx
     app: nginx
 – from:
   – podSelector:
         app: foo

Once you apply this configuration, only pods with label app: foo can talk to the pods with the label app: nginx. For a more detailed tutorial, see the Kubernetes documentation.

Example: restricting traffic between all pods by default

If you specify the spec.podSelector field as empty, the set of pods the network policy matches to all pods in the namespace, blocking all traffic between pods by default. In this case, you must explicitly create network policies whitelisting all communication between the pods.

You can enable a policy like this by applying the following manifest in your Kubernetes cluster:

kind: NetworkPolicy
 name: default-deny


Other Network Policy features

In addition to the previous examples, you can make the Network Policy API enforce more complicated rules:

  • Egress network policies: Introduced in Kubernetes 1.8, you can restrict your workloads from establishing connections to resources outside specified IP ranges.
  • IP blocks support: In addition to using podSelector/namespaceSelector, you can specify IP ranges with CIDR blocks to allow/deny traffic in ingress or egress rules.
  • Cross-namespace policies: Using the ingress.namespaceSelector field, you can enforce Network Policies for particular or for all namespaces in the cluster. For example, you can create privileged/system namespaces that can communicate with pods even though the default policy is to block traffic.
  • Restricting traffic to port numbers: Using the ingress.ports field, you can specify port numbers for the policy to enforce. If you omit this field, the policy matches all ports by default. For example, you can use this to allow a monitoring pod to query only the monitoring port number of an application.
  • Multiple ingress rules on a single policy: Because spec.ingress field is an array, you can use the same NetworkPolicy object to give access to different ports using different pod selectors. For example, a NetworkPolicy can have one ingress rule giving pods with the kind: monitoring label access to port 9000, and another ingress rule for the label app: foo giving access to port 80, without creating an additional NetworkPolicy resource.

Learn more

Using RBAC, Generally Available in Kubernetes v1.8

By | Blog

Editor’s note: this post is part of a series of in-depth articles on what’s new in Kubernetes 1.8. Today’s post comes from Eric Chiang, software engineer, CoreOS, and SIG-Auth co-lead.

Kubernetes 1.8 represents a significant milestone for the role-based access control (RBAC) authorizer, which was promoted to GA in this release. RBAC is a mechanism for controlling access to the Kubernetes API, and since its beta in 1.6, many Kubernetes clusters and provisioning strategies have enabled it by default.

Going forward, we expect to see RBAC become a fundamental building block for securing Kubernetes clusters. This post explores using RBAC to manage user and application access to the Kubernetes API.

Granting access to users

RBAC is configured using standard Kubernetes resources. Users can be bound to a set of roles (ClusterRoles and Roles) through bindings (ClusterRoleBindings and RoleBindings). Users start with no permissions and must explicitly be granted access by an administrator.

All Kubernetes clusters install a default set of ClusterRoles, representing common buckets users can be placed in. The “edit” role lets users perform basic actions like deploying pods; “view” lets a user observe non-sensitive resources; “admin” allows a user to administer a namespace; and “cluster-admin” grants access to administer a cluster.

$ kubectl get clusterroles
NAME            AGE
admin           40m
cluster-admin   40m
edit            40m
# …


view            40m


ClusterRoleBindings grant a user, group, or service account a ClusterRole’s power across the entire cluster. Using kubectl, we can let a sample user “jane” perform basic actions in all namespaces by binding her to the “edit” ClusterRole:

$ kubectl create clusterrolebinding jane –clusterrole=edit –user=jane
$ kubectl get namespaces –as=jane
NAME          STATUS    AGE
default       Active    43m
kube-public   Active    43m
kube-system   Active    43m
$ kubectl auth can-i create deployments –namespace=dev –as=jane


RoleBindings grant a ClusterRole’s power within a namespace, allowing administrators to manage a central list of ClusterRoles that are reused throughout the cluster. For example, as new resources are added to Kubernetes, the default ClusterRoles are updated to automatically grant the correct permissions to RoleBinding subjects within their namespace.

Next we’ll let the group “infra” modify resources in the “dev” namespace:

$ kubectl create rolebinding infra –clusterrole=edit –group=infra –namespace=dev
rolebinding “infra” created

Because we used a RoleBinding, these powers only apply within the RoleBinding’s namespace. In our case, a user in the “infra” group can view resources in the “dev” namespace but not in “prod”:

$ kubectl get deployments –as=dave –as-group=infra –namespace dev
No resources found.
$ kubectl get deployments –as=dave –as-group=infra –namespace prod
Error from server (Forbidden): deployments.extensions is forbidden: User “dave” cannot list deployments.extensions in the namespace “prod”.

Creating custom roles

When the default ClusterRoles aren’t enough, it’s possible to create new roles that define a custom set of permissions. Since ClusterRoles are just regular API resources, they can be expressed as YAML or JSON manifests and applied using kubectl.

Each ClusterRole holds a list of permissions specifying “rules.” Rules are purely additive and allow specific HTTP verb to be performed on a set of resource. For example, the following ClusterRole holds the permissions to perform any action on “deployments”, “configmaps,” or “secrets”, and to view any “pod”:

kind: ClusterRole
 name: deployer
– apiGroups: [“apps”]
 resources: [“deployments”]
 verbs: [“get”, “list”, “watch”, “create”, “delete”, “update”, “patch”]
– apiGroups: [“”] # “” indicates the core API group
 resources: [“configmaps”, “secrets”]
 verbs: [“get”, “list”, “watch”, “create”, “delete”, “update”, “patch”]
– apiGroups: [“”] # “” indicates the core API group
 resources: [“pods”]
 verbs: [“get”, “list”, “watch”]

Verbs correspond to the HTTP verb of the request, while the resource and API groups refer to the the resource being referenced. Consider the following Ingress resource:

apiVersion: extensions/v1beta1
kind: Ingress
 name: test-ingress
   serviceName: testsvc
   servicePort: 80

To POST the resource, the user would need the following permissions:

– apiGroups: [“extensions”] # “apiVersion” without version
 resources: [“ingresses”]  # Plural of “kind”
 verbs: [“create”]         # “POST” maps to “create”


Roles for applications

When deploying containers that require access to the Kubernetes API, it’s good practice to ship an RBAC Role with your application manifests. Besides ensuring your app works on RBAC enabled clusters, this helps users audit what actions your app will perform on the cluster and consider their security implications.

A namespaced Role is usually more appropriate for an application, since apps are traditionally run inside a single namespace and the namespace’s resources should be tied to the lifecycle of the app. However, Roles cannot grant access to non-namespaced resources (such as nodes) or across namespaces, so some apps may still require ClusterRoles.

The following Role allows a Prometheus instance to monitor and discover services, endpoints, and pods in the “dev” namespace:

kind: Role
 name: prometheus-role
 namespace: dev
– apiGroups: [“”] # “” refers to the core API group
 Resources: [“services”, “endpoints”, “pods”]
 verbs: [“get”, “list”, “watch”]

Containers running in a Kubernetes cluster receive service account credentials to talk to the Kubernetes API, and service accounts can be targeted by a RoleBinding. Pods normally run with the “default” service account, but it’s good practice to run each app with a unique service account so RoleBindings don’t unintentionally grant permissions to other apps.To run a pod with a custom service account, create a ServiceAccount resource in the same namespace and specify the `serviceAccountName` field of the manifest.

apiVersion: apps/v1beta2 # Abbreviated, not a full manifest
kind: Deployment
 name: prometheus-deployment
 namespace: dev
 replicas: 1
     – name: prometheus
       image: prom/prometheus:v1.8.0
       command: [“prometheus”, “-config.file=/etc/prom/config.yml”]
   # Run this pod using the “prometheus-sa” service account.
   serviceAccountName: prometheus-sa
apiVersion: v1
kind: ServiceAccount
 name: prometheus-sa
 namespace: dev

Get involved

Development of RBAC is a community effort organized through the Auth Special Interest Group, one of the many SIGs responsible for maintaining Kubernetes. A great way to get involved in the Kubernetes community is to join a SIG that aligns with your interests, provide feedback, and help with the roadmap.

About the author

Eric Chiang is a software engineer and technical lead of Kubernetes development at CoreOS, the creator of Tectonic, the enterprise-ready Kubernetes platform. Eric co-leads Kubernetes SIG Auth and maintains several open source projects and libraries on behalf of CoreOS.

It Takes a Village to Raise a Kubernetes

By | Blog

Editor’s note: this post is part of a series of in-depth articles on what’s new in Kubernetes 1.8, written by Jaice Singer DuMars from Microsoft.


Each time we release a new version of Kubernetes, it’s enthralling to see how the community responds to all of the hard work that went into it. Blogs on new or enhanced capabilities crop up all over the web like wildflowers in the spring. Talks, videos, webinars, and demos are not far behind. As soon as the community seems to take this all in, we turn around and add more to the mix. It’s a thrilling time to be a part of this project, and even more so, the movement. It’s not just software anymore.

When circumstances opened the door for me to lead the 1.8 release, I signed on despite a minor case of the butterflies. In a private conversation with another community member, they assured me that “being organized, following up, and knowing when to ask for help” were the keys to being a successful lead. That’s when I knew I could do it — and so I did.

From that point forward, I was wrapped in a patchwork quilt of community that magically appeared at just the right moments. The community’s commitment and earnest passion for quality, consistency, and accountability formed a bedrock from which the release itself was chiseled.

The 1.8 release team proved incredibly cohesive despite a late start. We approached even the most difficult situations with humor, diligence, and sincere curiosity. My experience leading large teams served me well, and underscored another difference about this release: it was more valuable for me to focus on leadership than diving into the technical weeds to solve every problem.

Also, the uplifting power of emoji in Slack cannot be overestimated.An important inflection point is underway in the Kubernetes project. If you’ve taken a ride on a “startup rollercoaster,” this is a familiar story. You come up with an idea so crazy that it might work. You build it, get traction, and slowly clickity-clack up that first big hill. The view from the top is dizzying, as you’ve poured countless hours of life into something completely unknown. Once you go over the top of that hill, everything changes. Breakneck acceleration defines or destroys what has been built.

In my experience, that zero gravity point is where everyone in the company (or in this case, project) has to get serious about not only building something, but also maintaining it. Without a commitment to maintenance, things go awry really quickly. From codebases that resemble the Winchester Mystery House to epidemics of crashing production implementations, a fiery descent into chaos can happen quickly despite the outward appearance of success. Thankfully, the Kubernetes community seems to be riding our growth rollercoaster with increasing success at each release.

As software startups mature, there is a natural evolution reflected in the increasing distribution of labor. Explosive adoption means that full-time security, operations, quality, documentation, and project management staff become necessary to deliver stability, reliability, and extensibility. Also, you know things are getting serious when intentional architecture becomes necessary to ensure consistency over time.

Kubernetes has followed a similar path. In the absence of company departments or skill-specific teams, Special Interest Groups (SIGs) have organically formed around core project needs like storage, networking, API machinery, applications, and the operational lifecycle. As SIGs have proliferated, the Kubernetes governance model has crystallized around them, providing a framework for code ownership and shared responsibility. SIGs also help ensure the community is sustainable because success is often more about people than code.

At the Kubernetes leadership summit in June, a proposed SIG architecture was ratified with a unanimous vote, underscoring a stability theme that seemed to permeate every conversation in one way or another. The days of filling in major functionality gaps appear to be over, and a new era of feature depth has emerged in its place.

Another change is the move away from project-level release “feature themes” to SIG-level initiatives delivered in increments over the course of several releases. That’s an important shift: SIGs have a mission, and everything they deliver should ultimately serve that. As a community, we need to provide facilitation and support so SIGs are empowered to do their best work with minimal overhead and maximum transparency.

Wisely, the community also spotted the opportunity to provide safe mechanisms for innovation that are increasingly less dependent on the code in kubernetes/kubernetes. This in turn creates a flourishing habitat for experimentation without hampering overall velocity. The project can also address technical debt created during the initial ride up the rollercoaster. However, new mechanisms for innovation present an architectural challenge in defining what is and is not Kubernetes. SIG Architecture addresses the challenge of defining Kubernetes’ boundaries. It’s a work in progress that trends toward continuous improvement.

This can be a little overwhelming at the individual level. In reality, it’s not that much different from any other successful startup, save for the fact that authority does not come from a traditional org chart. It comes from SIGs, community technical leaders, the newly-formed steering committee, and ultimately you.

The Kubernetes release process provides a special opportunity to see everything that makes this project tick. I’ll tell you what I saw: people, working together, to do the best they can, in service to everyone who sets out on the cloud native journey.

kubeadm v1.8 Released: Introducing Easy Upgrades for Kubernetes Clusters

By | Blog

Editor’s note: this post is part of a series of in-depth articles on what’s new in Kubernetes 1.8

Since its debut in September 2016, the Cluster Lifecycle Special Interest Group (SIG) has established kubeadm as the easiest Kubernetes bootstrap method. Now, we’re releasing kubeadm v1.8.0 in tandem with the release of Kubernetes v1.8.0. In this blog post, I’ll walk you through the changes we’ve made to kubeadm since the last update, the scope of kubeadm, and how you can contribute to this effort.

Security first: kubeadm v1.6 & v1.7

Previously, we discussed planned updates for kubeadm v1.6. Our primary focus for v1.6 was security. We started enforcing role based access control (RBAC) as it graduated to beta, gave unique identities and locked-down privileges for different system components in the cluster, disabled the insecure `localhost:8080` API server port, started authorizing all API calls to the kubelets, and improved the token discovery method used formerly in v1.5. Token discovery (aka Bootstrap Tokens) graduated to beta in v1.8.

In number of features, kubeadm v1.7.0 was a much smaller release compared to v1.6.0 and v1.8.0. The main additions were enforcing the Node Authorizer, which significantly reduces the attack surface for a Kubernetes cluster, and initial, limited upgrading support from v1.6 clusters.

Easier upgrades, extensibility, and stabilization in v1.8

We had eight weeks between Kubernetes v1.7.0 and our stabilization period (code freeze) to implement new features and to stabilize the upcoming v1.8.0 release. Our goal for kubeadm v1.8.0 was to make it more extensible. We wanted to add a lot of new features and improvements in this cycle, and we succeeded.Upgrades along with better introspectability. The most important update in kubeadm v1.8.0 (and my favorite new feature) is one-command upgrades of the control plane. While v1.7.0 had the ability to upgrade clusters, the user experience was far from optimal, and the process was risky.


Now, you can easily check to see if your system can handle an upgrade by entering:
$ kubeadm upgrade plan

This gives you information about which versions you can upgrade to, as well as the health of your cluster.

You can examine the effects an upgrade will have on your system by specifying the –dry-run flag. In previous versions of kubeadm, upgrades were essentially blind in that you could only make assumptions about how an upgrade would impact your cluster. With the new dry run feature, there is no more mystery. You can see exactly what applying an upgrade would do before applying it.

After checking to see how an upgrade will affect your cluster, you can apply the upgrade by typing:

$ kubeadm upgrade apply v1.8.0

This is a much cleaner and safer way of performing an upgrade than the previous version. As with any type of upgrade or downgrade, it’s a good idea to backup your cluster first using your preferred solution.


Self-hosting in this context refers to a specific way of setting up the control plane. The self-hosting concept was initially developed by CoreOS in their bootkube project. The long-term goal is to move this functionality (currently in an alpha stage) to the generic kubeadm toolbox. Self-hosting means that the control plane components, the API Server, Controller Manager and Scheduler are workloads themselves in the cluster they run. This means the control plane components can be managed using Kubernetes primitives, which has numerous advantages. For instance, leader-elected components like the scheduler and controller-manager will automatically be run on all masters when HA is implemented if they are run in a DaemonSet. Rolling upgrades in Kubernetes can be used for upgrades of the control plane components, and next to no extra code has to be written for that to work; it’s one of Kubernetes’ built-in primitives!

Self-hosting won’t be the default until v1.9.0, but users can easily test the feature in experimental clusters. If you test this feature, we’d love your feedback!

You can test out self-hosting by enabling its feature gate:

$ kubeadm init –feature-gates=SelfHosting=true


We’ve added some new extensibility features. You can delegate some tasks, like generating certificates or writing control plane arguments to kubeadm, but still drive the control plane bootstrap process yourself. Basically, you can let kubeadm do some parts and fill in yourself where you need customizations. Previously, you could only use kubeadm init to perform “the full meal deal.” The inclusion of the kubeadm alpha phase command supports our aim to make kubeadm more modular, letting you invoke atomic sub-steps of the bootstrap process.

In v1.8.0, kubeadm alpha phase is just that: an alpha preview. We hope that we can graduate the command to beta as kubeadm phase in v1.9.0. We can’t wait for feedback from the community on how to better improve this feature!


Along with our new kubeadm features, we’ve also made improvements to existing ones. The Bootstrap Token feature that makes `kubeadm join` so short and sweet has graduated from alpha to beta and gained even more security features.

If you made customizations to your system in v1.6 or v1.7, you had to remember what those customizations were when you upgraded your cluster. No longer: beginning with v1.8.0, kubeadm uploads your configuration to a ConfigMap inside of the cluster, and later reads that configuration when upgrading for a seamless user experience.

The first certificate rotation feature has graduated to beta in v1.8, which is great to see. Thanks to the Auth Special Interest Group, the Kubernetes node component kubelet can now rotate its client certificate automatically. We expect this area to improve continuously, and will continue to be a part of this cross-SIG effort to easily rotate all certificates in any cluster.

Last but not least, kubeadm is more resilient now. kubeadm init will detect even more faulty environments earlier, and time out instead of waiting forever for the expected condition.

The scope of kubeadm

As there are so many different end-to-end installers for Kubernetes, there is some fragmentation in the ecosystem. With each new release of Kubernetes, these installers naturally become more divergent. This can create problems down the line if users rely on installer-specific variations and hooks that aren’t standardized in any way. Our goal from the beginning has been to make kubeadm a building block for deploying Kubernetes clusters and to provide kubeadm init and kubeadm join as best-practice “fast paths” for new Kubernetes users. Ideally, using kubeadm as the basis of all deployments will make it easier to create conformant clusters.

kubeadm performs the actions necessary to get a minimum viable cluster up and running. It only cares about bootstrapping, not about provisioning machines, by design. Likewise, installing various nice-to-have addons by default like the Kubernetes Dashboard, some monitoring solution, cloud provider-specific addons, etc. is not in scope. Instead, we expect higher-level and more tailored tooling to be built on top of kubeadm, that installs the software the end user needs.

v1.9.0 and beyond

What’s in store for the future of kubeadm?

Planned features

We plan to address high availability (replicated etcd and multiple, redundant API servers and other control plane components) as an alpha feature in v1.9.0. This has been a regular request from our user base.

Also, we want to make self-hosting the default way to deploy your control plane: Kubernetes becomes much easier to manage if we can rely on Kubernetes’ own tools to manage the cluster components.

Promoting kubeadm adoption and getting involved

The kubeadm adoption working group is an ongoing effort between SIG Cluster Lifecycle and other parties in the Kubernetes ecosystem. This working group focuses on making kubeadm more extensible in order to promote adoption of it for other end-to-end installers in the community. Everyone is welcome to join. So far, we’re glad to announce that kubespray started using kubeadm under the hood, and gained new features at the same time! We’re excited to see others follow and make the ecosystem stronger.

kubeadm is a great way to learn about Kubernetes: it binds all of Kubernetes’ components together in a single package. To learn more about what kubeadm really does under the hood, this document describes kubeadm functions in v1.8.0.

If you want to get involved in these efforts, join SIG Cluster Lifecycle. We meet on Zoom once a week on Tuesdays at 16:00 UTC. For more information about what we talk about in our weekly meetings, check out our meeting notes. Meetings are a great educational opportunity, even if you don’t want to jump in and present your own ideas right away. You can also sign up for our mailing list, join our Slack channel, or check out the video archive of our past meetings. Even if you’re only interested in watching the video calls initially, we’re excited to welcome you as a new member to SIG Cluster Lifecycle!

If you want to know what a kubeadm developer does at a given time in the Kubernetes release cycle, check out this doc. Finally, don’t hesitate to join if any of our upcoming projects are of interest to you!

Thank you,
Lucas Käldström
Kubernetes maintainer & SIG Cluster Lifecycle co-lead
Weaveworks contractor

Five Days of Kubernetes 1.8

By | Blog

Kubernetes 1.8 is live, made possible by hundreds of contributors pushing thousands of commits in this latest releases.

The community has tallied more than 66,000 commits in the main repo and continues rapid growth outside of the main repo, which signals growing maturity and stability for the project. The community has logged more than 120,000 commits across all repos and 17,839 commits across all repos for v1.7.0 to v1.8.0 alone.

With the help of our growing community of 1,400 plus contributors, we issued more than 3,000 PRs and pushed more than 5,000 commits to deliver Kubernetes 1.8 with significant security and workload support updates. This all points to increased stability, a result of our project-wide focus on maturing process, formalizing architecture, and strengthening Kubernetes’ governance model.

While many improvements have been contributed, we highlight key features in this series of in-depth posts listed below. Follow along and see what’s new and improved with storage, security and more.

Day 1: 5 Days of Kubernetes 1.8
Day 2: kubeadm v1.8 Introduces Easy Upgrades for Kubernetes Clusters
Day 3: Kuberentes v.1.8 Retrospective: It Takes a Village to Raise a Kubernetes
Day 4: Using RBAC, Generally Available in Kubernetes v1.8
Day 5: Enforcing Network Policies in Kubernetes


  • Post questions (or answer questions) on Stack Overflow
  • Join the community portal for advocates on K8sPort
  • Follow us on Twitter @Kubernetesio for latest updates
  • Connect with the community on Slack
  • Get involved with the Kubernetes project on GitHub

BlaBlaCar: Turning to Containerization to Support Millions of Rideshares

By | Blog

For the 40 million users of BlaBlaCar, it’s easy to find strangers headed in the same direction to share rides and costs. You can even choose how much “bla bla” chatter you want from a long-distance ride mate.

Behind the scenes, though, the infrastructure was falling woefully behind the rider community’s exponential growth. Founded in 2006, the company hit its current stride around 2012. “Our infrastructure was very traditional,” says Infrastructure Engineer Simon Lallemand. “When you’re thinking about doubling the number of servers, you start thinking, ‘What should I do to be more efficient?’ The answer is not to hire more and more people just to deal with the servers and installation.”

Instead, BlaBlaCar began its cloud-native journey but wasn’t sure which route to take. “Going into the cloud meant we had to make a lot of changes in our application work, and we were just not ready to make the switch from on premise to the cloud,” says Lallemand. They wanted to keep the great performance they got on bare metal, so they didn’t want to go to virtualization on premise.

The solution: containerization. This was early 2015 and containers were still relatively new. “It was a bold move at the time,” says Lallemand. “We decided to go with containers and with CoreOS Container Linux as an abstraction for this hardware. It seemed future-proof to go with containers because we could see what companies were already doing with containers.”

Next, they needed to choose a runtime for the containers, but “there were very few deployments in production at that time,” says Lallemand. They experimented with Docker but decided to go with rkt. Lallemand explains that for BlaBlaCar, it was “much simpler to integrate things that are on rkt.” At the time,  the project was still pre-v1.0, so “we could speak with the developers of rkt and give them feedback. It was an advantage.” Plus, he notes, rkt was very stable, even at this early stage.

The first project the team decided to containerize was the database, which required creating some of their own tools. At the same time, the company was working to migrate its entire platform to containers with a deadline set for Christmas 2015. With all the work being done in parallel, BlaBlaCar was able to get about 80 percent of its production into containers by its deadline with live traffic running on containers during December. (It’s now at 100 percent.)

In the middle of that peak season for carpooling, everything worked well. “The biggest impact that we had was for the deployment of new services,” says Lallemand. “Before using containers, we had to first deploy a new server and create configurations with Chef. It would take sometimes a day, sometimes two, just to create a new service. And with all the tooling that we made around the containers, copying a new service is a matter of minutes. So it’s really a huge gain.”

In order to meet their self-imposed deadline, one of the decisions they made was to not do any “orchestration magic” for containers in the first production alignment. “But at some point you want to give more autonomy to the developer team,” Lallemand says. “We also realized that we don’t want to be the single point of contact for developers when they want to launch new services.” By the summer of 2016, they found their answer in Kubernetes, which had just begun supporting rkt implementation. They also started using Prometheus, as they were looking for “service-oriented monitoring that could be updated nightly.” Production on Kubernetes began in December 2016. “We like to do crazy stuff around Christmas,” he adds with a laugh.

BlaBlaCar now has about 3,000 pods, with 1200 of them running on Kubernetes. Lallemand leads a “foundations team” of 25 members who take care of the networks, databases and systems for about 100 developers.

When Lallemand speaks to other European companies curious about BlaBlaCar’s cloud-native journey, he tells them to come along for the ride: “It’s such a pleasure to deal with the infrastructure that we have today compared to what we had before.”


To learn more about what’s ahead for BlaBlaCar with Kubernetes and rkt, check out this in-depth case study.

Interested in more Kubernetes content? Get on the CNCF newsletter list for more Kubernetes information and updates.

Pear Deck: Infrastructure for a Growing Edtech Startup

By | Blog

With the speed befitting a startup, Pear Deck delivered its first prototype to customers within three months of incorporating.

As a former high school math teacher, CEO Riley Eynon-Lynch felt an urgency to provide a tech solution to classes where instructors struggle to interact with every student in a short amount of time. “Pear Deck is an app that students can use to interact with the teacher all at once,” he says. “When the teacher asks a question, instead of just the kid at the front of the room answering again, everybody can answer every single question. It’s a huge fundamental shift in the messaging to the students about how much we care about them and how much they are a part of the classroom.”

Eynon-Lynch and his partners quickly built a Javascript web app on Google’s web app development platform Firebase, and launched the minimum viable product [MVP] on Heroku “because it was fast and easy,” he says. “We made everything as easy as we could.”

But once it launched, the user base began growing steadily at a rate of 30 percent a month. “Our Heroku bill was getting totally insane,” Eynon-Lynch says. But even more crucially, as the company hired more developers to keep pace, “we outgrew Heroku. We wanted to have multiple services and the deploying story got pretty horrendous. We were frustrated that we couldn’t have the developers quickly stage a version. Tracing and monitoring became basically impossible.”

The team began looking around for another solution, and finally decided in early 2016 to start moving the app from Heroku to Docker containers running on Google Container Engine, orchestrated by Kubernetes and monitored with Prometheus.

Once the team started porting its Heroku apps into Kubernetes, which was “super easy,” he says, the impact was immediate. “Before, to make a new version of the app meant going to Heroku and reconfiguring 10 new services, so basically no one was willing to do it, and we never  staged things,” he says. “Now we can deploy our exact same configuration in lots of different clusters in 30 seconds. We stage all the time now, and everyone stopped talking about how cool it is because it’s become invisible how great it is.”

Along with Kubernetes came Prometheus. After Helm installed Prometheus, “We started getting a graph of the health of all our Kubernetes nodes and pods immediately. I think we were pretty hooked at that point,” Eynon-Lynch says. With Pear Deck’s specific challenges—traffic through Firebase as well as government firewalls—Prometheus was a game-changer. “We didn’t even realize how stressed out we were about our lack of insight into what was happening with the app,” Eynon-Lynch says. Before, when a customer would report that the app wasn’t working, the team had to manually investigate the problem without knowing whether customers were affected all over the world, or whether Firebase was down, and where.

To help solve that problem, the team wrote a script that pings Firebase from several different geographical locations, and then reports the responses to Prometheus in a histogram. Plus, Prometheus has allowed Pear Deck to build alarms for business goals. One measures the rate of successful app loads and goes off if the day’s loads are less than 90 percent of the loads from seven days before. “That gives us a lot of confidence,” says Eynon-Lynch, “and we at least know that people are still logging into the app and using it all the time.”

For a spry startup that’s continuing to grow rapidly—and yes, they’re hiring!—Pear Deck is notably satisfied with how its infrastructure has evolved in the cloud native ecosystem. “Usually I have some angsty thing where I want to get to the new, better technology,” says Eynon-Lynch, “but in terms of the cloud, Kubernetes and Prometheus have so much to offer.”

To learn more about what’s ahead for Pear Deck with Kubernetes and Prometheus, check out this in-depth case study.

Interested in more Kubernetes content? Get on the CNCF newsletter list for more Kubernetes information and updates.

Buffer: Making Deployments Easy for a Small, Distributed Team

By | Blog

Dan Farrelly uses a carpentry analogy to explain the problem his company, Buffer, began having as its team of developers grew over the past few years.

“If you’re building a table by yourself, it’s fine,” the company’s architect says. “If you bring in a second person to work on the table, maybe that person can start sanding the legs while you’re sanding the top. But when you bring a third or fourth person in, someone should probably work on a different table.” Needing to work on more and more different tables led Buffer on a path toward microservices and containerization made possible by Kubernetes.

Since around 2012, Buffer had already been using Elastic Beanstalk, the orchestration service for deploying infrastructure offered by Amazon Web Services. “We were deploying a single monolithic PHP application, and it was the same application across five or six environments,” says Farrelly. “We were very much a product-driven company. It was all about shipping new features quickly and getting things out the door, and if something was not broken, we didn’t spend too much time on it.”

Figure 1: Dan Farrelly presenting How Kubernetes Was the Secret Sauce in Our Globally Distributed Team’s Transition to Microservices at CloudNativeCon + KubeCon North America 2016

But things came to a head in 2016. With the growing number of committers on staff, Farrelly and Buffer’s then-CTO, Sunil Sadasivan, decided it was time to re-architect and rethink their infrastructure. Some of the company’s team was already successfully using Docker in their development environment. They wanted to go further with Docker, and the next step was looking at options for orchestration.

Kubernetes suited Buffer’s needs. “We wanted to have the kind of liquid infrastructure where a developer could create an app and deploy it and scale it horizontally as necessary,” says Farrelly. “We quickly used some scripts to set up a couple of test clusters, we built some small proof-of-concept applications in containers, and we deployed things within an hour. We had very little experience in running containers in production. It was amazing how quickly we could get a handle on it.”

Kubernetes provided a powerful solution for one of the company’s most distinguishing characteristics: their remote team that’s spread across a dozen different time zones. “We really wanted something where anybody could get a grasp of the system early on and utilize it, and not have to worry that the deploy engineer is asleep,” Farrelly says.

For Buffer’s engineers, it’s an exciting process. “Every time we’re deploying a new service, we need to figure out: OK, what’s the architecture? How do these services communicate? What’s the best way to build this service?” Farrelly says. “And then we use the different features that Kubernetes has to glue all the pieces together. It’s enabling us to experiment as we’re learning how to design a service-oriented architecture.”

Figure 2: Buffer Senior Software Engineer Harrison Harnisch presenting Load Testing Kubernetes: How to Optimize Your Cluster Resource Allocation in Production at CloudNativeCon + KubeCon Europe 2017

At this point, the team at Buffer can’t imagine running their infrastructure any other way—and they’re happy to spread the word. “If you want to run containers in production, with nearly the power that Google uses internally, this is a great way to do that,” Farrelly says. “Pick a couple of things, roll it out, kick the tires on this for a couple of months and see how much it can handle. You start learning a lot this way.”

For a closer look at Buffer’s decision-making process as it rolled cloud native infrastructure to help its distributed team deploy faster, check out this in-depth case study.

Interested in more content like this? We can curate and deliver relevant articles just like this, directly to you, once a month in our CNCF newsletter. Get on the list here.