Guest post by Ben Hirschberg, VP R&D and Co-Founder of ARMO

Can you keep a secret? Hope so, because in this blog, I reveal the secrets of Kubernetes secrets. First, I dive into the mechanics of Kubernetes secrets and then move to how to protect them.

Kubernetes secrets are the native resources for storing and managing sensitive data, like passwords, cloud access keys, or authentication tokens. You need to distribute this information across your Kubernetes clusters and protect it at the same time. When sending your password to each node in your cluster, it’s critical to ensure that only authorized entities—users, services, or workloads—are able to access it.

Kubernetes Secrets

The building blocks of Kubernetes computation are pods made up of containers. You can bake your sensitive passwords into container images or configure them as part of pod definitions. The more secure and Kubernetes-native approach uses secret objects and introduces them in pod specifications (e.g., a file or environment variable).

In the next section, I introduce the protection layers Kubernetes offers.

Protections

Secrets are native Kubernetes resources, and Kubernetes provides a basic set of protection layers for them. These protection measures are developed over time and can be grouped as follows:

These protection measures ensure that secrets are separated from other Kubernetes resources, accessed and stored securely. Still, Kubernetes isn’t a bulletproof security system and comes with some risks.

Risks

When it comes to secrets, the potential risks are identified and open for mitigation by third-party tools and extensions. The risks can be grouped as follows:

Next up, I cover open-source, cloud-specific, and more inclusive tools to mitigate the risks above and protect Kubernetes secrets.

Protecting Kubernetes Secrets

There are various tools and strategies that can take your Kubernetes secrets security to the next level. In this section, I cover the available tools as well as their benefits and drawbacks to help you choose the right one (or more than one) for designing your clusters.

Cloud Key Management Systems

Cloud providers like GCP and AWS have their own cloud key management systems (KMS), a centralized cloud service through which you can create and manage keys to perform cryptographic operations. These systems can also serve as an additional security layer for the encryption of Kubernetes secrets at the application layer. 

The greatest benefit of KMS, of course, is its ability to protect against attackers who have gained access to a copy of etcd. It can, however, be a little tricky to integrate, so be sure to first do your research on how to integrate KMS into your cluster and ensure that it conforms with your security operations. On the other hand, there are two major drawbacks to KMS: firstly, you need to change your application code to work with KMS providers; secondly, it relies on authenticating the node, so if the attackers are already on the node, they can access your data easily.

Sealed Secrets

Sealed secrets help mitigate the risks related to CaC and leaking secrets into source-code repositories by encrypting secrets locally into a format that is safe to store and publish. When the cluster needs to use the secret, it is decrypted only by the controller running in the cluster. 

This approach requires installing a controller to the cluster and a client tool called kubeseal to the local workstation. If you only care about protecting your secret objects as a file, it’s pretty straightforward to install and operate. However, you need to be careful about sealed secrets since they only protect against theft from the source code repository. For instance, if you check out your keys and secrets to your workstation during installation, they will be in danger.

Helm Secrets Plugin

Helm is a useful tool for installing complex applications to clusters, including their configuration and sensitive data. However, the problem of leaking secret values into source code repositories still applies to Helm charts. Operations teams maintain the Helm charts in source code repositories with value files in separate or even in the same location. 

Helm secrets is a Helm plugin for encrypting secrets through Mozilla’s open source SOPS project. It is also an extendable platform that supports external key management systems like Google Cloud KMS and AWS KMS. If you’re deploying your applications with Helm and storing sensitive data in your values, you’ll want to be sure to secure your Helm chart values. Similar to sealed secrets, the Helm secrets plugin only protects the sensitive data in the source code repository. It does not provide any protection for the storage of secrets in the Kubernetes API or etcd. 

To sum up, Kubernetes secrets are the cloud-native way of storing and managing sensitive information in the cloud. While Kubernetes offers some protection, there are still many open risks to consider. By incorporating one or more tools in your operations stack, you can mitigate these risks.

ARMO empowers DevOps, DevSecOps, and security teams with the confidence that their Kubernetes deployments are protected and that every workload, container, and microservice is born and remains secure. Visit our site to learn more, schedule a demo, or take the first step to adding seamless protection to your Kubernetes cluster by trying ARMO out for free.

About the Author 

Ben is the VP R&D and Co-Founder of ARMO. Hands-on experience in cloud architectures and DevOps platforms, Ben has more than 15 years of software security experience, starting at the early days of Cloud Native technology.