Diversity Scholarship Series: Making the Microservices, Kubernetes, and Cloud Native Connection

By | Blog

CNCF offered diversity scholarships to developers and students to attend KubeCon + CloudNativeCon China 2018. In this post, Emmelyn Wang shares her Diversity Scholarship experience of visiting China for the first time as a person of Chinese descent, valuable conference takeaways + much more. Anyone interested in applying for the CNCF diversity scholarship to attend KubeCon + CloudNativeCon Europe 2019 in Barcelona May 20-23, can submit an application here — applications are due March 11th — and anyone interested in attending KubeCon + CloudNativeCon China 2019 in Shanghai June 24-26, can submit an application here — applications are due April 15th.

By Emmelyn Wang, Principal Services Architect and API Strategist at Axway

I’m based in Texas and was fortunate enough to attend the first KubeCon + CloudNativeCon conference in China after’s DevRelCon London and speaking at API the Docs.


Before KubeCon + CloudNativeCon, I…

  • Felt overwhelmed about where to start and intimidated about contributing to open source software projects.
  • Joined a core team of a technical specification (AsyncAPI which is related to OAS) to learn more about contributing as a technical writer with specific API and microservices knowledge and experience.

After KubeCon + CloudNativeCon, I…

  • Felt grateful to visit China for the first time in my life, as a person of Chinese descent and as a recipient of the CNCF’s Diversity Scholarship.
  • Feel confident about continuing contributions to the open source software community.
  • Feel knowledgeable about CNCF documentation sprints because of my experience at the event, meeting maintainers — including Dr. Brad Topol — contributors, and reading the O’Reilly book I received, Deploying Kubernetes in the Enterprise.

My Top 3 valuable conference takeaways are…

  1. Attending KubeCon + CloudNativeCon showed me how microservices and native and hybrid cloud architecture scale at an enterprise level.
  2. Making contributions accessible is key to reaching developers around the world asynchronously. I appreciated talks like this session that are foundational to inviting and involving new contributors.
  3. My global view of container, cloud native, and open source collaboration is now complete. The conference experience provided a glimpse into local companies from China and the amazing things they’re doing—Alibaba, Huawei, Tencent, Rancher, Caicloud, Daocloud,, and more. This portion has always been the missing piece for me since I have only worked in North America—understanding how APAC is building and contributing. It was exciting to hear the announcement about Harbor, the first CNCF project founded in China moving into Incubation status while at the conference.

Through the event, I also met my coworker — David Wang, Principal Services Architect I, Axway — who is based in Singapore. I asked him about his experience:

“This is the first time for me to join KubeCon + CloudNativeCon. For me, this event is a good opportunity to learn how these open source projects, like Kubernetes and Istio, are used by different companies and the best practices of implementing Kubernetes and Istio in different clouds and on prem environments.

From my perspective, the lessons they have learned and shared when migrating to Kubernetes are also very helpful. More importantly, I noticed that Kubernetes and Istio are becoming the actual standard for container orchestration and microservices.

As an API architect, if we want to be more confident when getting involved in the discussions with customers or prospects, we need to understand how Kubernetes and Istio work and the underlying technologies. A good understanding of container and microservices techniques will become one of the key factors to the success of API projects, no matter they are on-prem or cloud.”

Special thanks to the CNCF Diversity Scholarship committee for choosing me. I will continue sharing my story and encourage technical communicators, software developers, and architects to contribute to open source projects. For example, I can share specific instructions and ideas with groups including WWCode Women Who Code, Write the Docs, API the Docs, and Developer Conferences around the world.

Here is a brief photo gallery of my journey to the first KubeCon + CloudNativeCon China + feel free to connect with me via Medium or Twitter!

Kubernetes Day India Schedule Announced

By | Blog

We are pleased to announce the schedule for Kubernetes Day India, our inaugural event taking place on March 23 in Bengaluru, India, at Infosys Limited, our Venue sponsor. Additional sponsors include Platinum sponsor, DigitalOcean; Gold sponsors, InfraCloud and OpenEBS and Silver sponsors Bluemeric, CloudYuga, Platformer, and Portworx.

Kubernetes Day is a single day, single track event that brings together local and international experts to engage developers of all levels interested in Kubernetes and related cloud native technologies. CNCF is excited to bring this new event to regions with large numbers of developers who might not necessarily travel to KubeCon + CloudNativeCon events in Europe, China, and North America. Stay tuned for details on future #KubernetesDay events taking place around the world.

Talks will range from introductory-level to technical, given by speakers from diverse companies driving the technologies and the end users deploying them, including:

  • Keynote from Liz Rice,Technology Evangelist, Aqua Security @lizrice
  • Noobernetes 101: Top 10 Questions We Get From New K8s Users – Neependra Khare, CloudYuga Technologies @neependra
  • How to Secure Your Kubernetes Clusters – Cindy Blake, GitLab @cblake2000
  • Using Kubernetes API Effectively with Golang – Vishal Biyani, @vishal_biyani
  • Making Cloud Native Computing Universal and Sustainable – Dee Kumar, CNCF @deesprinter
  • Kubernetes for Java Developers – Arun Gupta, Amazon Web Services @arungupta
  • Building a PaaS for Robotics with Kubernetes – Dhananajay Sathe, Rapyuta Robotics @d_sathe
  • How to Contribute to Kubernetes – Nikhita Raghunath, Loodse @TheNikhita

Registration is open and we are offering discounted tickets in order to bring the community together. A program committee led by conference chairperson Liz Rice of Aqua Security selected a mix of local and international Kubernetes experts to present. Until March 1, early-bird tickets are available for 500 INR (which is 7 USD). In addition, Meetup members will receive a 20% discount for the event! We are also offering diversity scholarships for applicants from underrepresented and/or marginalized groups. The deadline to apply for scholarships is Friday, February 22, 2019.

If you are not based in India, visa requirements are detailed on the website.

Join the Community

Kubernetes Day India is a great opportunity to join the Kubernetes community, learn from and meet others and get involved! A strong open source project needs a diverse community and we hope you will join to further the education and advancement of cloud native computing.

Interested in getting involved with the Kubernetes community before the event? We encourage your participation and contributions!

We are looking forward to seeing you in Bengaluru next month!

KubeCon + CloudNativeCon North America 2018 Conference Transparency Report: A Record-Breaking CNCF Event

By | Blog

KubeCon + CloudNativeCon North America 2018 was a huge success with record-breaking registrations, attendance, sponsorships, and co-located events. Out of 8,000 attendees, 73% were first-time KubeCon-ers, highlighting massive growth and new interest in CNCF and cloud native technologies.  

The KubeCon + CloudNativeCon North America 2018 conference transparency report

Key Takeaways

  • Attendance grew by 90% from last year’s KubeCon event in Austin.
  • The three day conference offered 318 sessions, including breakouts, lightning talks, BoFs, tutorials and Maintainer Track sessions.
  • 1,824 companies participated, including 196 CNCF member companies.
  • Feedback from attendees was overwhelmingly positive, with an overall average rating of 4.35 / 5 of those surveyed.
  • The top two reasons people attended the event were for networking (34%) and to attend breakout sessions (36%).
  • The event was made up of 11% of women, up 1.5% from Austin in 2017 – while 40% of the keynote speakers were female and 14% of the track sessions were led by women.
  • With attendees from 66 different countries, the $300,000 in diversity scholarship funding provided by CNCF along with Aspen Mesh, MongoDB, Twistlock, Two Sigma and VMware resulted in 144 applicants receiving travel and/or conference registration compensation.
  • Almost 100 media and analysts attended the event, a 72% increase from Austin in 2017

KubeCon Gets Bigger and Bigger!

Taking place December 10- 13 in Seattle, KubeCon + CloudNativeCon North America 2018 was the biggest KubeCon ever. The event was so popular that it not only sold out to 8,000 attendees, but the waitlist surpassed another 1,000. Thanks to the live stream, 1,580 people who could not attend tuned into the the keynote sessions.

Attendees expressed interest in all CNCF projects, with Kubernetes (72%), Prometheus (42%), Helm (32%) and Envoy (28%), leading the charge.

Women Shine at KubeCon

KubeCon + CloudNativeCon North America 2018 was a great event for the women in our community! Female attendance grew by 1.5% over the previous year, and there was a huge presence of women as keynote and track session speakers. In fact, 40% of the keynotes were given by women. While at the event, attendees had the opportunity to attend activities to promote diversity of all kinds and the EmpowHER evening event, sponsored by Comcast, attracted more than 200 participants.

A Growing List of End Users

As cloud native technologies are increasingly adopted in the enterprise, end users were abound at KubeCon + CloudNativeCon North America 2018! With an acceptance rate of roughly 2x that of previous events, attendees had the opportunity to hear compelling insights from end user companies implementing cloud native technologies, including BlackRock, Fairfax Media, T-Mobile, USA Today, Intuit and more!

Co-Located Events

A record-breaking 26 co-located events took place during KubeCon + CloudNativeCon North America 2018 from a number of different companies. Three of these events – Envoycon, Observability Practitioners Summit and Kubernetes Contributor Summit – were put on by CNCF and sold out far in advance!

What Reporters and Analysts are Saying

More than three dozen news announcements from our member companies resulted in many insightful and compelling articles from media and analysts. Here are some highlights:

Save the Dates for 2019!

Registration is open for KubeCon + CloudNativeCon Europe 2019, which will be taking place in Barcelona from May 20-23.

KubeCon + CloudNativeCon + Open Source Summit China will be happening from June 24-26. Registration is now open and the call for speakers and proposals are due February 22.  

And finally, we will be in sunny San Diego for KubeCon + CloudNativeCon North America from November 18-21.

Hope to see you there!

Diversity Scholarship Series: Why Not?

By | Blog

CNCF offered diversity scholarships to developers and students to attend KubeCon + CloudNativeCon China 2018. In this post, Dennis Salamanca Farafonov shares what sets KubeCon + CloudNativeCon apart and why you should apply to become a Diversity Scholarship recipient. Anyone interested in applying for the CNCF diversity scholarship to attend KubeCon + CloudNativeCon Europe 2019 in Barcelona May 20-23, can submit an application here — applications are due March 11th — and anyone interested in attending KubeCon + CloudNativeCon China 2019 in Shanghai June 24-26, can submit an application here — applications are due April 15th.

By Dennis Salamanca Farafonov, a technology solutions professional, open source specialist, and Certified Kubernetes Application Developer/Administrator with Azure

Before KubeCon + CloudNativeCon, I have never attended an IT conference before. Although, I have always been obsessed with them. The idea of thousands of people gathered in the same place by a single technology in such a massive event, tickles my interest in an extraordinary way. To go and hear from the experts about their experiences or attended the talk of one of your “IT Heroes”, is almost a religious experience. I know that many of us in IT feel the same way; and I’m sure that if any of you get the opportunity to go to an event like this, you will do it in a heartbeat, no questions asked.

It’s just amazing how our field of work has evolved in the last two decades to become one of the most important and influential ones in the world. It is no surprise how many people are attracted to technology nowadays and why this type of event has become so massive. Today, it’s almost as if there is a conference every month. But many of these are hosted by big multi-billion-dollar companies, whose purpose is to evangelize their products or to perform big announcement of the “What’s Next” in their own portfolios. Don’t get me wrong here, I’m not saying that this is something bad — not at all. These big companies are why we are where we are, and why “Cons” have become mainstream in the tech industry. I had to mention the big company “Cons” to talk to you about KubeCon + CloudNativeCon. Oh man, KubeCon + CloudNativeCon is a whole different beast.

Just to say that KubeCon + CloudNativeCon revolves around a completely open source project that has reached a level of importance that we haven’t seen since the Linux Kernel; with a hundred or more companies, big or small, gathered to share their experiences, collaborate and work together to shape the next generation of IT, leaves me with no words to describe it. To bring people together and collaborate across different backgrounds, is the core of what open source is, and KubeCon + CloudNativeCon represents that in a whole new level. And we can’t ignore that it is only thanks to foundations like the Cloud Native Computing Foundation and The Linux Foundation that events like KubeCon + CloudNativeCon are possible.

It gives me the chills to only think about how big Kubernetes has become in the last two years. KubeCon + CloudNativeCon and Kubernetes are changing our way of thinking about how we do software and information technology, the industry has awakened and opened its eyes to see that it’s through collaboration, networking and open code that we can reach success in not only in a faster way but in a more reliable and exponential way.

Diversity is the basis of what we as an open source community are and it’s through programs like CNCF’s Diversity Scholarship initiative, that we enable KubeCon + CloudNativeCon to represent the values of who we are and empower those who without this tool wouldn’t be able to experience, receive and give back to the community first hand in an event like this.

If you are into open source, Kubernetes and all things cloud native, as a Scholarship recipient, I encourage you to apply — just go ahead and do it. The only regret you’ll have is not to have done it. The experience of being there, the people you are going to meet and the knowledge you come back with is invaluable.

How Uber Monitors 4,000 Microservices

By | Blog

With 4,000 proprietary microservices and a growing number of open source systems that needed to be monitored, by late 2014 Uber was outgrowing its usage of Graphite and Nagios for metrics. They evaluated several technologies, including Atlas and OpenTSDB, but the fact that a growing number of open source systems were adding native support for the Prometheus Metrics Exporter format tipped the scales in that direction.

Uber found with its use of Prometheus and M3, Uber’s storage costs for ingesting metrics became 8.53x more cost effective per metric per replica. The team estimates that setting up monitoring systems in Uber data centers for its Advanced Technologies Group was 4x faster than it would have been under the previous process.

Make sure to check out the full case study!

Diversity Scholarship Series: Bringing along Kubernetes experience from Shanghai to Nepal

By | Blog

CNCF offered diversity scholarships to developers and students to attend KubeCon + CloudNativeCon China 2018. In this post, our scholarship recipient Raksha Roy, an Enterprise Resource Planning Associate from Nepal, shares her experience attending sessions and meeting the community. Anyone interested in applying for the CNCF diversity scholarship to attend KubeCon + CloudNativeCon Europe 2019 in Barcelona May 20-23, can submit an application here — applications are due March 11th — and anyone interested in attending KubeCon + CloudNativeCon China 2019 in Shanghai June 24-26, can submit an application here — applications are due April 15th.

By Raksha Roy, an Enterprise Resource Planning Associate from Nepal

Blog originally posted on Medium

While Kubernetes is dominantly rising today as the world’s most popular container orchestration tool, with large community as well as its competitors, I barely heard about it in my country, Nepal. I am thankful towards MOOCs where I could start learning Kubernetes from. Further exposure was provided by The Linux Foundation by providing “Diversity Scholarship” to attend KubeCon + CloudNativeCon Conference, Shanghai, 2018. I would not have been capable to talk about Kubernetes back in my country among my peers had it not been that opportunity I received to attend the conference. The series of learning and networking that started after getting the scholarship acceptance email have not just enriched my knowledge, but also enhanced my capacity for professional growth. Moreover, interacting with diverse professionals helped me get involved in the community easily.

There were 2500 attendees including CNCF project maintainers, open source community members, technology enthusiasts and other users from very popular, Alibaba and eBay. I used for years without knowing that it now runs the world’s largest Kubernetes cluster in production. There is not just one topic you get to learn about from this conference! It’s just not learning about Kubernetes increasing resource utilization, or largest companies moving to cloud native, or shooting up of clouds usages all over the world, or containers replacing VMs widely, or learning new architectures and algorithms. It is limitless opportunity to see the world gather at a single spot and pick numerous things to learn. grabbed the 2018 End user award

Keynotes, especially the ones in the mornings at Kubecon CloudNativeCon conference fuel up with great energies for the remaining day, involving speakers from some of the tech giants, like, Aqua Security, Cloud Native Computing Foundation, VMware, Google, Alibaba Cloud, Microsoft, IBM, Rancher Labs, Huawei, Lyft, Tencent Cloud and GitLab.

The day grew intense with discussions, presentations and deep dives on projects from CNCF, Fluent Bit, SIG (Apps, Cloud Provider, Autoscaling, Service Catalog), Kubernetes, Rook, Containerd, Falco, gRPC, Prometheus, Jaeger, Helm,Harbor, and more!

“Harbor is not only the first project donated by VMware to CNCF, but it is also the first Chinese program developed in the Chinese open source community donated to CNCF”- CNCF.

Throughout the day, the vendor showcase had much more to offer than freebies, cute goodies and free t-shirts. Demos and other illustrations provided an opportunity to closely interact with the presenters and learn about specific products in detail.

The platform has provided an excellent learning experiences, both in terms of professional and personal development. There were fun performances too at the day end, to relish upon rich Chinese culture, providing an opportunity for global participants to have some insights into local events.

There was a popular spot at the conference, where career opportunities could be awaiting for many. One could pin a job or find one, posted on the board.

Another complimentary highlight of the conference was an hour of Tai Chi class.

There are several challenges growing up and working in a developing country, majorly related to infrastructures and learning platforms. I have been highly privileged to receive the diversity scholarship, where I could learn and share knowledge among global communities. I am thankful as I can now initiate to bring back home Kubernetes, in my work network, and share global experience in a local platform. Overall, the KubeCon + CloudNativeCon conference has instilled in me more confidence and provided more room to grow. Thank you Kubecon + CloudNativeCon!

New Year, New TOC 🌟

By | Blog

The CNCF TOCcharged with defining and maintaining the technical vision for CNCF; approving new projects within the scope for CNCF set by the Governing Board, and creating a conceptual architecture for these projects; aligning projects, removing or archiving projects; accepting feedback from end user committee and map to projects; aligning interfaces to components under management (code reference implementations before standardizing); and defining common practices to be implemented across CNCF projectsis crucial for driving neutral consensus within this community.

As the committee’s inaugural three-year terms were coming to a close, we recently kicked off elections for the following six Technical Oversight Committee (TOC) slots that were set to open up this month:

After receiving nominations from members, the following six community leaders were selected by our Governing Board to serve two-year, staggered terms, effective immediately 🎉

  • Alexis Richardson (Weaveworks) → CNCF has been transformative for everyone I know. I am just very excited to be part of the next stage and help the user community grow with these superb projects.”
  • Brendan Burns (Microsoft)“The last three years have seen ‘cloud native’ go from ‘what does that mean?’ to ‘required reading.’ As I look forward to the next three years of the CNCF, I’m excited to move from supporting and incubating projects, to helping them mature, scale and work together to deliver on this fundamental transition in the way the people build computing applications.”
  • Joe Beda (VMware) “The CNCF sits at the center of an exciting inflection point as a new crop of revolutionary open source projects redefine IT. The CNCF TOC helps to guide this process. As a Kubernetes founder and technology leader at VMware, I’m excited and humbled to play my part in making sure that the CNCF both picks effective projects and is, in turn, effective in supporting and boosting those projects.”
  • Kelsey Hightower (Google)“As a newly elected member of the TOC I’m looking forward to representing the voice of the community and ensuring our projects continue to exceed their expectations.”
  • Matt Klein (Lyft)Cloud native OSS is the future of distributed computing, and the CNCF is a key player in the space. I’m honored to join the TOC to help guide which projects the foundation accepts, ensure that accepted projects are nurtured so they have the opportunity to thrive, and most importantly, clarify a diverse technology landscape for end users.”
  • Xiang Li (Alibaba) “The CNCF has been driving the cloud innovation through open source technologies since its founding. CNCF projects like Kubernetes, Envoy, Prometheus, etcd make cloud native computing accessible and portable for a wide range of users. As a developer of CNCF projects and technology leader at Alibaba Cloud, I am committed and passionate to help CNCF with projects direction and growth, listening to and representing end users needs, as well as growing the Cloud Native community especially in Asia-Pacific.”

In addition to these six new members, Jeff Brewer (Intuit) was also selected to join the TOC as an End User TOC representative.

We’d love to extend a big thank you to our outgoing TOC members – Jonathan, Bryan, Camille, Benjamin & Ken – for their years of commitment to safeguarding CNCF’s technical vision and growing this community into a neutral home for the ecosystem’s most promising technologies!

New to the community and/or curious about how our TOC operates? Join us for one of the committee’s public, bi-monthly calls – find more info here ☑️

Enterprise Leaders’ Protips for Scavenger Hunting Through the Cloud Native Tool Weeds

By | Blog

By R.D. Danes, a staff writer for SiliconANGLE

This article was written and produced as part of the Orate project, which helps organizations tell technical stories

Aren’t enterprises lucky the cloud-native ecosystem is growing so lavishly? A whole universe of well-honed tools is expanding before them. The means with which to modernize their infrastructure and applications are at their fingertips. The problem? They’re lost in a herd of tools that might work for some organizations, but not their own. At the Kubecon + CloudNativeCon North America 2018 conference in Seattle, Washington, last month, enterprise leaders gathered to share tips on scavenging the best tools from the bunch.

A panel titled, “Avoiding the Weeds in the Cloud Native Landscape,” brought together four professionals toiling in the enterprise-IT trenches. Moderator Priyanka Sharma, director of alliances for GitLab and contributor to the OpenTracing project, picked panelists’ brains for tricks to cloud-native tool shopping. They shared real-world successes and failures in procuring, trying, returning and swapping tools.

Why cloud native?

Two solid rules for IT teams wading into cloud native stood out in the discussion: 1) Do thorough research upfront, and 2) don’t hesitate to chuck a selection that doesn’t pan out.

Before embarking on a tool-hoarding spree to cure all your app-performance woes, make sure to have a reality check on what’s possible, advised Melissa Chapman, director of IT for PaaS and configuration management at CVS Health.

“If you have a bad piece of code or a bad application, and you think a tool is going to fix it, you are so wrong,” she said.

Enterprises need to keep their endgame top of mind. Why cloud native? Do they want to speed up application development and deployment? Give their developers more elbow room to innovate on impulse? Increase the agility of their DevOps teams? Do they know precisely how cloud-native technologies are going to help them achieve these goals?

“Tooling is a means to an end,” said Brendan Aye, director of platform architecture for T-Mobile.

“You don’t see Wall Street Journal articles about the great tool sets some companies put out there.”

With all the hype around fresh technology, it’s easy to forget that it’s useful only in as far as it improves business outcomes. What might work wonders for one business may flop fantastically in another. It might not be compatible older technologies, or staffers may lack the skills needed to work with it.

That said, picking useful tools isn’t entirely down to trial and error. Companies can certainly define goals and establish some standards that tools must meet before they give them a test run.

Introducing the unofficial AEIOU cloud-native standards body

Anyone remember learning the vowels?

Jasmine James, senior systems engineer for Delta Air Lines, and her colleagues repurposed the old “AEIOU” song. They turned it into an acronym and use it to size up cloud-native tools. It goes like this: Applicability, Enterprise-Ready, Integration, Overhead and Usefulness.

The most important element of all is usefulness, James said. Even if other businesses have had a ton of success with a tool, Delta won’t grab it until it’s sure it can get some juice out of it. “We want to make sure you can actually use them in the right way.”

Use-case cure for transformation anxiety attack

Usefulness is most easily determined on the basis of some well-defined use case. The business knows the problem, knows what hasn’t worked in the past and what might finally fix it.

An incremental approach to adopting cloud-native technologies and modernizing applications is much wiser than boiling the ocean, according to Matt Klein, software engineer for Lyft and creator of CNCF project Envoy.

“That’s really what I recommend to everyone on their journey is to really think about what problems are being faced and try to do them incrementally. Like, there’s no Big Bang solution,” he said. “Adopting a single technology is not going to transform a business overnight, but a single tool could vastly improve one use case or application. A series of simple steps like that could eventually result in a transformed business.”

Reel in, throw back, repeat for best catch

What if a tool doesn’t work? What if it’s a total disaster? Easy — throw it out and try something new.

“There’s dozens of tools for any single use case,” Aye said. “Perhaps CNCF’s project Prometheus 1.0 failed miserably in your company, but Prometheus 2.0 might deliver everything the first version lacked”.

Chapman agreed that rapid iteration is the royal road to finding the best fit in cloud-native tooling.

CVS wanted to digitize its receipts so that customers could view them on smartphones. It experimented with cloud-native technologies, failed, continued and finally succeeded.

“We started with some tools that we thought were going to help us and didn’t,” she said.

“And so we shifted very quickly —  but it ended up being fantastic.”

To integrate or not to integrate?

When a company finally finds the tools that work for it, it may soon realize there are too many of them to manage. What then? Should it opt for tools that cluster features together?

“Perhaps not”, Aye said. “I don’t want to name vendors, but there are some logging tools now that started integrating metrics and some metrics tools that started integrating logging — and I feel like that’s a good way to be bad at both those things.”

Best-of-breed point solutions are usually preferable to mediocre medleys. The cloud-native ecosystem has work to do integrating tools for easier management without watering down their features.

We shall see how much progress it has made in May when the CNCF comes together again in Barcelona, Spain, for KubeCon + CloudNativeCon EU 2019 (call for proposals close January 19th).

9 Kubernetes Security Best Practices Everyone Must Follow

By | Blog

By Connor Gilbert, product manager at StackRox 

Last month, the Kubernetes ecosystem was shaken by the discovery of the first major security flaw in Kubernetes, the world’s most popular container orchestrator. The vulnerability – CVE-2018-1002105 – enables attackers to compromise clusters via the Kubernetes API server, allowing them run code to perform malicious activity such as installing malware, etc.

Earlier this year, Tesla suffered a  complex cryptocurrency mining malware infection caused by a misconfiguration in the Kubernetes console. The attackers exploited the fact that the particular Kubernetes console wasn’t password protected, allowing them to access one of the pods that included access credentials for Tesla’s larger AWS environment.

As organizations accelerate their adoption of containers and container orchestrators, they will need to take necessary steps to protect such a critical part of their compute infrastructure. To help in this endeavor, check out these nine Kubernetes security best practices, based on customer input, you should follow to help protect your infrastructure.

  1. Upgrade to the Latest Version

New security features — and not just bug fixes — are added in every quarterly update, and to take advantage of them, we recommend you run the latest stable version. The very best thing to do is to run the latest release with its most recent patches, especially in light of the discovery of CVE-2018-1002105. Upgrades and support can become more difficult the farther behind you fall, so plan to upgrade at least once per quarter. Using a managed Kubernetes provider can make upgrades very easy.

  1. Enable Role-Based Access Control (RBAC)

Control who can access the Kubernetes API and what permissions they have with Role- Based Access Control (RBAC). RBAC is usually enabled by default in Kubernetes 1.6 and beyond (later for some managed providers), but if you have upgraded since then and haven’t changed your configuration, you’ll want to double-check your settings. Because of the way Kubernetes authorization controllers are combined, you must both enable RBAC and disable legacy Attribute-Based Access Control (ABAC).

Once RBAC is being enforced, you still need to use it effectively. Cluster-wide permissions should generally be avoided in favor of namespace-specific permissions. Avoid giving anyone cluster admin privileges, even for debugging — it is much more secure to grant access only as needed on a case-by-case basis.

You can explore the cluster roles and roles using `kubectl get clusterrolebinding` or `kubectl get rolebinding –all-namespaces`. Quickly check who is granted the special “cluster-admin” role; in this example, it’s just the “masters” group:

If your application needs access to the Kubernetes API, create service accounts individually and give them the smallest set of permissions needed at each use site. This is better than granting overly broad permissions to the default account for a namespace.

Most applications don’t need to access the API at all; `automountServiceAccountToken` can be set to “false” for these.

  1. Use Namespaces to Establish Security Boundaries

Creating separate namespaces is an important first level of isolation between components. We find it’s much easier to apply security controls such as Network Policies when different types of workloads are deployed in separate namespaces.

Is your team using namespaces effectively? Find out now by checking for any non-default namespaces:

  1. Separate Sensitive Workloads

To limit the potential impact of a compromise, it’s best to run sensitive workloads on a dedicated set of machines. This approach reduces the risk of a sensitive application being accessed through a less-secure application that shares a container runtime or host. For example, a compromised node’s kubelet credentials can usually access the contents of secrets only if they are mounted into pods scheduled on that node — if important secrets are scheduled onto many nodes throughout the cluster, an adversary will have more opportunities to steal them.

You can achieve this separation using node pools (in the cloud or on-premises) and Kubernetes namespaces, taints, tolerations, and other controls.

  1. Secure Cloud Metadata Access

Sensitive metadata, such as kubelet admin credentials, can sometimes be stolen or misused to escalate privileges in a cluster. For example, a recent Shopify bug bounty disclosure detailed how a user was able to escalate privileges by confusing a microservice into leaking information from the cloud provider’s metadata service. GKE’s metadata concealment feature changes the cluster deployment mechanism to avoid this exposure, and we recommend using it until it is replaced with a permanent solution. Similar countermeasures may be needed in other environments.

  1. Create and Define Cluster Network Policies

Network Policies allow you to control network access into and out of your containerized applications. To use them, you‘ll need to make sure that you have a networking provider that supports this resource; with some managed Kubernetes providers such as Google Kubernetes Engine (GKE), you‘ll need to opt in. (Enabling network policies in GKE will require a brief rolling upgrade if your cluster already exists.) Once that’s in place, start with some basic default network policies, such as blocking traffic from other namespaces by default.

If you are running in Google Container Engine, you can check whether your clusters are running with policy support enabled:

  1. Run a Cluster-wide Pod Security Policy

A Pod Security Policy sets defaults for how workloads are allowed to run in your cluster. Consider defining a policy and enabling the Pod Security Policy admission controller — instructions vary depending on your cloud provider or deployment model. As a start, you could require that deployments drop the NET_RAW capability to defeat certain classes of network spoofing attacks.

  1. Harden Node Security

You can follow these three steps to improve the security posture on your nodes:

  • Ensure the host is secure and configured correctly. One way to do so is to check your configuration against CIS Benchmarks; many products feature an autochecker that will assess conformance with these standards automatically.
  • Control network access to sensitive ports. Make sure that your network blocks access to ports used by kubelet, including 10250 and 10255. Consider limiting access to the Kubernetes API server except from trusted networks. Malicious users have abused access to these ports to run cryptocurrency miners in clusters that are not configured to require authentication and authorization on the kubelet API server.
  • Minimize administrative access to Kubernetes nodes. Access to the nodes in your cluster should generally be restricted — debugging and other tasks can usually be handled without direct access to the node.
  1. Turn on Audit Logging

Make sure you have audit logs enabled and are monitoring them for anomalous or unwanted API calls, especially any authorization failures — these log entries will have a status message “Forbidden.” Authorization failures could mean that an attacker is trying to abuse stolen credentials. Managed Kubernetes providers, including GKE, provide access to this data in their cloud console and may allow you to set up alerts on authorization failures.

Looking Ahead

Follow these recommendations for a more secure Kubernetes cluster. Remember, even after you follow these tips to configure your Kubernetes cluster securely, you will still need to build security into other aspects of your container configurations and their runtime operations. As you improve the security of your tech stack, look for tools that provide a central point of governance for your container deployments and deliver continuous monitoring and protection for your containers and cloud-native applications.

Monitoring Kubernetes, part 1: the challenges + data sources

By | Blog

Originally published here by Sean Porter, CTO of Sensu

Our industry has long been relying on microservice-based architecture to deliver software faster and safer. The advent and ubiquity of microservices naturally paved the way for container technology, empowering us to rethink how we build and deploy our applications. Docker exploded onto the scene in 2013, and, for companies focusing on modernizing their infrastructure and cloud migration, a tool like Docker is critical to shipping applications quickly, at scale.

Container shipBut, with that speed comes challenges — containers introduce a non-trivial level of complexity when it comes to orchestration. Enter Kubernetes: an open source container-orchestration system for automating deployment, scaling, and management of containerized applications — the Kubernetes control plane isthe command and control for your infrastructure. Originally launched by Google in 2014, Kubernetes is now maintained by the Cloud Native Computing Foundation (which, incidentally, Google helped form in order to place Kubernetes within the CNCF, to make sure it’d stay free and competitive). If you’re using Docker to containerize your applications, then you’re most certainly using Kubernetes for orchestration. (There are certainly other orchestrators, such as Docker Swarm and Apache Mesos, but Kubernetes has emerged as the leader in container orchestration.)

In the first part of this series, I’ll cover the challenges and the main data sources for monitoring Kubernetes. Later on, I’ll dive deeper into monitoring Kubernetes and Docker deployments, with real-world examples drawing on the data sources outlined below.

Kubernetes monitoring: the challenges

Kubernetes makes it a lot easier for teams to manage containers — scheduling and provisioning them while maintaining a desired state, automatically. A core value prop is it serves as a common platform — Kubernetes can deploy your applications wherever they run, whether that’s AWS, GCP, Azure, or bare metal. Again, with all that power and automation come challenges, especially when it comes to keeping an eye on performance. No matter the size of your deployment, you still need to know how many available resources you have in that deployment, as well as knowing the health of your deployed applications and containers. Just as microservices led us to rethink how we build our applications, Kubernetes requires we change our traditional approach to monitoring — the dynamic nature of container orchestration insists on a subsequently dynamic approach to monitoring.

Here are the challenges as I see them:

  • In this new dynamic era, your applications are constantly moving.
  • Before Kubernetes, it was non trivial to have applications distributed across multiple clouds (public and private, as well as different cloud providers). Now that it’s easy to distribute applications, we have a new set of problems.
  • Much like the move from monolith to microservice architecture, adopting Kubernetes means there are many, smaller pieces to monitor.
  • You’ve heard about the merits of treating your infrastructure like cattle as opposed to pets. Kubernetes is the epitome of this livestock approach, making it easy to implement high volume and ephemeral infrastructure; just so, keeping track of of your Kubernetes pods and their containers via identifiers such as labels and annotations becomes mission critical.

Kubernetes monitoring: the data sources

Essentially, monitoring tools are collecting Kubernetes data from four sources:

  1. The Kubernetes hosts running the Kubelet. The Kubernetes hosts has limited resources, so it’s especially critical to monitor them. There are a number of ways to get data out of those hosts, but most commonly is to use the Prometheus node exporter to scrape data from the Kubernetes host and expose system resource telemetry data on an HTTP endpoint (such a CPU usage and memory).
  2. The Kubernetes process, AKA Kubelet metrics, which includes metrics for apiserver, kube-scheduler, and kube-controller-manager. These give you details on a Kubernetes node and the jobs it’s running.
  3. The Kubelet’s built-in cAdvisor. There’s a great summary here, but essentially the Kubelet ships with built-in support for cAdvisor, which collects, aggregates, processes, and exports metrics for your running containers. cAdvisor (which also has native support for Docker containers) gives you per-container usage, keeping track of resource isolation parameters and historical resource usage. Because Kubernetes is the control plane, it can designate how much memory is being used, and leverages cAdvisor to keep track.
  4. kube-state-metrics, which gives you information at the cluster level — a big picture view of what’s happening on your Kubernetes cluster, such as all the pods you have configured and their current state. kube-state-metrics hits all Kubernetes services and collects information on their current state, such as how many containers are running, how many are in a particular state, whether any are indicating that they’re unhealthy or that we’re at capacity, etc. From the README, kube-state-metrics “listens to the Kubernetes API server.”

Next up: container states and collecting data with Prometheus

If you’re keeping track at home, you may noticed you can monitor all four of these data sources with Prometheus. You may also have noticed that we’re only talking about monitoring Kubernetes but not the applications running on it (and, this may line up with everything you’ve heard about Kubernetes monitoring). In my next post, I’ll illustrate Kubernetes and Docker monitoring with Prometheus, discuss why it fits well within the Kubernetes ecosystem, and identify the gaps.