By Jenn Gile March 3, 2022

Guest post by Jenn Gile, Sr PMM, F5 NGINX

With each major shift in technology, we see new jobs with new titles emerge. Today, the biggest shift in infrastructure and application deployment is the widespread adoption of Kubernetes, and the latest job title we’re seeing is the “Kubernetes Operator.” This isn’t to be confused with the Kubernetes programmatic operator construct, which is a software extension used in Kubernetes to manage resources). Since the word “operator” has multiple meanings in Kubernetes, for the sake of simplicity we’ll use “Kubernetes Engineer” for the rest of this article.

This following graphic from the Kubernetes Gateway API site depicts the Kubernetes Engineer as a “cluster operator” who is responsible for managing Gateway functionality. The Gateway API is still in alpha, but this model applies equally to Kubernetes implementations using other ingress-egress options (such as the Ingress API).

Diagram flow shows a modern set of APIs for deploying L4 and L7 routing in Kubernetes
Image source: https://gateway-api.sigs.k8s.io/

The Kubernetes Engineer role is often covered by engineers with other titles, such as Site Reliability Engineer for web apps or System Administrator for more old-school IT departments exploring this brave new world. Cloud-native orgs and teams might make the role part of Platform Ops or Cloud Architect positions. But more and more, companies running modern apps or going down the cloud-native path need a dedicated Kubernetes expert.

This development harks back to the last big infrastructure shift, when virtualization replaced bare metal. Enterprises rushed to hire virtualization engineers to operate VMware and Citrix virtualization platforms. As in that transformation, there is a significant learning curve for Kubernetes Engineers. The vocabulary of Kubernetes is often confusing; a term also used in networking or apps may have a different meaning in the world of Kubernetes. At the same time, many newer terms — such as microservices — are closely tied to Kubernetes as the dominant deployment pattern for microservices. The service mesh, an entirely new piece of infrastructure designed to help teams manage and deliver microservices securely and reliably, is almost always associated with Kubernetes.

Going forward, almost every organization pursuing Kubernetes will need someone that fits the description of a Kubernetes Engineer. If they don’t have one already, they’ll almost certainly need one soon. This article will look at the role, what it means for your organization, and how you might shape the role to fit your needs.

Brian Ehlert and I were recently on the Day Two Cloud podcast where we talked about this new skillset. Give it a listen and check out the Q&A below for more on the in-and-outs of this emerging role.

https://packetpushers.net/podcast/day-two-cloud-136-the-role-and-responsibilities-of-a-kubernetes-operator-sponsored/

What Are the Basic Responsibilities of Kubernetes Engineer?

Kubernetes Engineers are tasked first and foremost with making sure that all things Kubernetes run well for their organization. Responsibilities usually include:

How Much Does the Size and Infrastructure of a Deployment Contribute to the Need for Kubernetes Engineer?

The extent of a Kubernetes Engineer role(s) depends on three factors:

If you’re a startup with a limited number of apps running on a small infrastructure footprint, “Kubernetes Engineer” is likely to be just one facet of someone’s job, possibly limited to deploying containers and apps on a managed Kubernetes service such as Google Kubernetes Engine. Such services abstract away concerns about the physical hardware and networking layer and do a pretty good job of handling scaling, basic security, and basic network management with minimal manual work.

But think about this for a bit before you decide that a part-time Kubernetes Engineer will suffice. The larger the infrastructure and the greater the number of moving parts, the more complex and time-consuming the job becomes. The diagram below depicts a very simple view. Broadly speaking, managing and operating Kubernetes on bare metal is far more complex than with a managed service. But using a managed service means zero visibility into the physical layer and it can add some latency. You might be surprised how often problems, conflicts, and performance issues can be traced to multi-tenant infrastructure. If you think noisy neighbors are bad in the cloud, imagine what they’re like in a managed Kubernetes service! Ditto for restarts of larger apps residing on big containers and higher capacity Kubernetes clusters.

Chart shows factors that contribute to the complexity of Kubernetes environments and management tasks

In addition to the infrastructure question, you need to consider how many clusters you’ll be running and how many pods and nodes are in each cluster. Multi-cluster Kubernetes is considerably more complicated because it requires additional customization of security, networking, and API and traffic management. It also requires managing and linking multiple Ingress controllers (more about that in a bit) and determining whether you need multiple clusters running for hot failover (expensive) or you can tolerate a bit of downtime while a backup cluster spins up from scratch. Observability and troubleshooting are also far less complicated with single clusters and a limited number of pods and services. As soon as you have to troubleshoot and observe performance of more than a handful of services running on more than a dozen pods, keeping an eye on everything becomes fairly taxing.

Do Kubernetes Engineers Deploy Apps or Only Manage Infrastructure?

This varies from place to place. In larger organizations, a Kubernetes Engineer typically focuses mostly on keeping Kubernetes environments safe and running well, rather than writing YAML manifests or overseeing application deployment. Typically working closely with the Platform Ops team (or as a member of that team), Kubernetes Engineers at larger orgs are more interested in setting up trusted catalogs of services and APIs or registries of containers vetted to work on their Kubernetes setups. In smaller organizations or where a team is more invested in running the guts of Kubernetes, Kubernetes Engineers may overlap with DevOps teams and be responsible for working with app developers to oversee deployments.

How Do You Know When You Need a Kubernetes Engineer?

As soon as you’re sure that Kubernetes will play a big part of your app infrastructure, you’ll  want to start thinking about bringing on a Kubernetes Engineer, or at minimum an app developer with some Kubernetes expertise and experience operating it in production. If you’re following the modern application design pattern of building a rock-solid Kubernetes cluster foundation before you start deploying apps – at F5 NGINX we call this pattern “cluster out” – then you’ll want a Kubernetes Engineer as soon as possible. There are many ways things can go wrong in Kubernetes, and problems with architecture, service design, and other flaws may not reveal themselves after you’ve “gone live” and the stakes are high. For this reason, developing your Kubernetes Engineering chops early in the adoption process is a wise move.

Is Kubernetes Certification Necessary?

While being a Certified Kubernetes Administrator (CKA) is definitely a plus, there are plenty of people who learn how to run a Kubernetes cluster the old-fashioned way with hard-earned experience and the scars to prove it. In fact, we believe that experience executing specific tasks is more useful than a CKA credential. The reason for this is two-fold:

The best of both worlds, of course, is experience and a certification. Pursuing a CKA after having spent time hands-on with Kubernetes is a great strategy.

Do Kubernetes Engineers Need Advanced Kubernetes Networking Expertise?

Not necessarily. Kubernetes Engineers need a strong grasp of Kubernetes networking concepts and experience configuring Ingress controllers, multi-cluster networking, and managing east-west and north-south traffic. But they don’t necessarily need experience with more advanced Container Network Interface (CNI) apps designed to provide something more like a network management and network security layer in Kubernetes. For the majority of Kubernetes use cases we see today, a CNI is overkill, making CNI skills a nice-to-have but not a need-to-have for Kubernetes Engineers.

Where Can I Learn More?

A solid foundation in Kubernetes networking concepts and tools is essential to success. That’s why we’re offering Microservices March 2022: Kubernetes Networking. It’s a free education program taking place March 7-31 that includes a series of events and self‑paced activities designed to take you from zero to hero. You can choose to complete the entire program or just the bits and pieces you need to fill in your knowledge gaps. The total time commitment is about 16 hours, spread across 4 weeks.

Four units progressively guide you through the essentials of Kubernetes networking:

Each unit includes:

Sign up today to get free access to the program – and don’t worry! Even if March 7th has already passed by the time you read this article, you can catch what you’ve missed on-demand.