Kubernetes is everywhere nowadays, so are the container images and fetching the images from a private registry is a norm because of N number of reasons including security, that being the topmost.
Recap
Just to give you a heads up before we dive further, let’s assume I run kubectl create deployment nginx –image nginx what do you think happens?
1. Using kubeconfig, kubectl first authenticates with the API server.
2. It then constructs a deployment object using the values which we provided like deployment name as nginx, image to be used as nginx and some default values and then sends a request to the API server.
3. API server then validates the request and the object definition.
4. This is further stored in etcd datastore.
5. Then the controller manager specifically deployment controller comes into the picture which creates a replica set.
6. Then the replica set creates a pod (this is where our nginx image will be used).
7. Then the scheduler assigns the pod to a node, where it will run based on multiple conditions. If interested please check out my previous article on what all things happens in the background: https://kubernetes.io/blog/2023/01/12/protect-mission-critical-pods-priorityclass/#resource-management-in-kubernetes
8. Once the node is identified, the kubelet running on the node is responsible for the management of the pod.
9. Once the pod is scheduled on a specific node, the kubelet looks for the image to be used and if it is already present.
- if that is present locally, it will use it.
- If not, it pulls the image from the registry.
- If the registry is public, there are no issues, it just pulls the image, runs the pods, and we are done.
When imagePullSecrets are used?
– Assume the registry is private, this is where imagePullSecrets come into play.
– Kubernetes will need the secrets to authenticate the registry, so that it can pull the image.
How it works?
– Firstly, we need to create a secret which contains the credentials and some details of our docker registry like server-name, username, password, email, etc.
– It can be created both imperative as well as declarative way.
– Quick note: The secret must be of a type kubernetes.io/dockerconfigjson or docker-registry based on the way you choose to create the secret.
– Once the secret is created, it must be attached to a pod so that it can use it.
– Just add below content in your pod/deployment/statefulset yaml files, the name of the secret for me is dockersec but you can change based on your requirements.
– Before pulling the image, the kubelet checks if the image is from a private registry. If so, it looks for imagePullSecrets in the spec.
– Without proper imagePullSecrets, the image pull will fail with ImagePullBackOff.
Problem
– Now assume you have hundreds of YAML files and hundreds of Kubernetes Object where we have to use this imagePullSecrets.
– Just for the sake of doing once, you thought to do it manually but just imagine there was a change in the name of the Kubernetes secret.
– Maybe because the organization now decides to streamline the naming convention of all the Kubernetes objects based on environments like prod/preprod/staging/etc.
– Or your org is now moving to a different registry or a different provider.
– Or the application is getting re-factored based on different namespaces or functions.
– There can be number of reasons and reapplying all these changes again is a tricky and tiresome task.
One way to look at it is, use of helm charts which can help us to manage all the Kubernetes objects and this kind of a change can be pretty straightforward provided the helm charts are structured correctly and using values file we can just update the name of the secret and then can upgrade the chart, and we are done.
But if in case we are not using a package manager like helm and we are just working with vanilla YAML files then for that as well there is a neat way to do it.
The NEAT way!
We can attach the imagePullSecrets with the serviceaccounts and where-ever the serviceaccount is used, it will automatically populate the imagePullSecrets in the respective object like Pods.
– Below is the current status of my Kubernetes objects, no secrets, a default serviceaccount and no imagePullSecrets updated in the pod.
– Create the docker-registry secret
– Patch the default serviceaccount to have the imagePullSecrets. It can be any serviceaccount based on your requirements.
– Or you can manually edit the serviceaccount as well and add below content.
– To see this in action now, delete the pod which is in failed state.
– Check the events of the pod, now it was successfully able to pull the image from the private registry.
– Interesting bit is imagePullSecrets are automatically added to the object.
Bonus Tip
– In case the pod is not in running status, it can be due to multiple reasons.
– Just describe the pod to see what is happening in the background.
– kubectl describe pod POD_NAME | awk ‘/^Events:/ {found=1; next} found’, it looks for the line starting with Events, once found prints the remaining lines.
– The logs clearly state, requested access to the resource is denied and in-turn the status of the pod is ImagePullBackOff.
References:
– Google.
– Kubernetes docs (https://kubernetes.io/docs/home/).
– kubectl explain.
– Linux man pages.
PS: This works with any flavor of Kubernetes: minikube, Elastic Kubernetes Service(EKS), Azure Kubernetes Service(AKS), Google Kubernetes Engine(GKE), etc.
I hope this will be helpful! Feel free to add your thoughts and experiences. Happy Learning
😄URL: A smarter way to use imagePullSecrets in Kubernetes Cluster using ServiceAccounts – DEV Community