Guest post originally published on the Oteemo blog by Tom Halligan


One of the challenges organizations face while adopting Kubernetes is providing Operations/Support personnel with the tools and training they need to support K8s deployments. Kubernetes adoption is often driven by development or engineering teams and those teams tend to use tooling that maps to their needs but may not map to break\fix support functions.

Support teams need to ramp up quickly on Kubernetes while continuing to support existing environments, therefore it is important that teams who provide tier 1/2 support of K8s have proper foundational training in Kubernetes. Management tools are of little use if the basics of K8s are not understood. Support teams need to focus on K8s architecture and basic principles through training or hands-on programs

I am touching on two tools, one that assists with K8s self-training and a second tool that is an excellent fit for break fix support of K8s.

Self Paced Training with K3d (

To assist in self learning there are single node k8s implementations (minikube) and cloud provided solutions that can be used with minimal expense (EKS, GKE etc) but the best, in my opinion, from light weight, quick deploy, and emulation of function standpoint is K3d a Dockerized Kubernetes cluster based on the K3s distribution maintained by Rancher.

Before we start we will need to install kubectl and Docker on your workstation see Appendix A below for links to install documents.

On a MAC run brew install k3d to install the tool on Linux run curl -s | bash

Once install is complete you can bring up your 3 node cluster

k3d cluster create somename -a 2

Run kubectl get nodes and you will see your cluster.

k3d-somename-worker-0   Ready 13s v1.16.2-k3s.1
k3d-somename-server     Ready master 14s v1.16.2-k3s.1
k3d-somename-worker-1   Ready 13s v1.16.2-k3s.1

You can now practice kubectl commands, simple deployments, tainting and labels for scheduling etc.

When you are done you can run the following to cleanup (But if you are following along let’s leave this cluster up to use with next tool)

k3d cluster delete somename

Operations/Support with K9s (

There are a number of management tools for Kubernetes clusters, but for my money the tool that most closely maps to day to day operations tasks is K9s.

So let’s do a quick example using the k3d cluster we setup above. First we will create a couple of deployments one good one bad


kubectl create deployment nginxgood –image=nginxkubectl create deployment nginxbad –image=nginy

Then lets install k9s and start it up

On a MAC you can run brew install derailed/k9s/k9s or you can download binary for Linux, Windows or MAC and put it in your path

Then run k9s

The UI will default to the pod view and we can see the pods from our two deployments, one set of pods is failing. You can select any of the pods and describe pod d, get logs l, shell into container running on pod s (You can select a pod hit enter and get a list of all containers in pod and shell into any of them), and even use CTRL-k to kill a pod.

k9s main screen

For our case pod describe d shows us what is wrong, a typo on the image name.

k9s describe pod

To switch between resource views you hit : which brings up command interpreter and then put in the short name for the resource (CTRL-a gives full list), in our case we are going to deployments so dp

k9s switch

We can then select the owner deployment and hit e to edit the deployment manifest (Will pop you into vi) we can then correct the typo and :wq!

k9s deployment
deply edit

Then we can switch back to pod view : po and see the deployment initiate new pods and terminate the old ones.

k9s good pods


The key to supporting any complex system is understanding, and that is where training/tooling is critical. Some implementations of applications in K8s can be wholly contained in CI/CD process that negates the need for interventionist support (When there is an issue you fix in code and redeploy). A significant number of organizations have not achieved that level of sophistication and K8s deployments will need support from techops resources, lest the Dev teams get overwhelmed with interruptive support work.

The above tools can assist support teams to rapidly learn the basics of K8s management and troubleshooting and quickly address level 1/2 issues that may occur.


Both k3d and k9s are great tools for development purposes also, with k9s especially have many great features (Custom kubectl plugins), they are not Operator specific, but great tools for a DevOPs organization to have on hand.

Appendix A:

kubectl install info

Docker Desktop on Windows

Docker Desktop on MAC