Guest post originally published on True‘s blog by Sebastiaan Kok
In Kubernetes secrets contain all kinds of sensitive information. For instance, database credentials or API keys. The term secrets management describes the centralised and secured management of these secrets. Sebastiaan Kok, Innovation Engineer at True, explains the importance of secret management and why this should be a key focus. He also points out how True handles secrets management itself and thus supports its customers with keeping their secrets safe.
“Secrets Management is a process where you manage secrets, like SSH keys, database credentials, certificates and API keys in a secure, centralised way. Access is controlled by role-based access control and policies”, says Sebastiaan.
Centralised management
Centralised management of secrets offers important advantages. Sebastiaan: “Especially when a secret is in use in multiple instances, for example at multiple moments in the CI/CD pipeline or an application, the management of it can be very complex. Think of changing or rotating a secret. It can involve a lot of work. Centralised secret management simplifies this.”
Secret management covers more than only guarding passwords. “Secret management concerns the management of all secrets within an organisation and centralised storage of it. Often, multiple teams work in one organisation, and each of them has its own secrets. Most of the time, everyone works in their own way with these secrets. Centralised management of it can be tricky and a challenge”, says Sebastiaan.
Secret management and Kubernetes
On the Kubernetes platform, adequate management of secrets is a key focus. “This is particularly a challenge if you work with GitOps”, Sebastiaan says. With GitOps, you configure Kubernetes deployments based on specifications stored in a Git repository. This repository forms a single source of truth.
“Secrets needed for configuring a Kubernetes deployment are not encrypted by design. It is comparable with storing a password in a plain text document, which is a very bad idea. You’ll need a way for working safely with secrets with GitOps.”
Working with multiple clusters
Secret management should also be a key focus when working with multiple Kubernetes clusters. “By design, Kubernetes saves secrets in a cluster. However, if you work with multiple clusters, secrets are also stored in multiple locations.”
Another use is management of secrets for environmental variables. “Think of CI/CD pipelines, for example when a pipeline needs to access a secured resource like SSH or a database server”, Sebastiaan says. You can also use secrets management for automation. “For instance, if you work with Ansible or Terraform. You’ll need secrets for those as well.”
The most important part: using a solution with strong encryption. This defines to a high level the safety of stored secrets. “A challenge can be configuring policies. Who has access to which specific secret? Configuring should also be a key focus.”
Base64 encoded
Kubernetes are solely base64 encoded. “In this coding process, binary data are converted to ASCII. Base64 can be decoded by everyone. In other words: you save passwords in plain text, which is, again, very unsafe.”
“Git is intended to share source code, for example with colleagues, external companies you work with, or sometimes a community when it’s about open source code. Saving secrets in plain text in code of Git is a big no go. Encoding is not encryption”, explains Sebastiaan.
Safely storing Kubernetes secrets
Among other, open source projects offer a way out for safely storing Kubernetes secrets. Think of Bitnami Sealed Secrets and KSOPS. Sealed Secrets is a way to encrypt and convert secrets to a Sealed Secret. You can safely save these in Git.
“When deploying a Sealed Secret in Kubernetes, it decrypts a so-called controller, who is included in the cluster of the Sealed Secret. This is converted in a regular Kubernetes secret”, says Sebastiaan. This way of working also has it disadvantages. “For the use of Sealed Secret, a developer also needs access to kubectl and need permissions to encrypt secrets. To change a current Sealed Secret a developer needs to dispose of unencrypted secrets locally. Besides preventing secrets stored on a local system, this solution doesn’t scale when working with a large development team.”
Vault
But there are more options available. “At True, we choose for a way where storing encrypted keys in Git isn’t necessary: Vault of HashiCorp. In a deployment, you’ll refer to a location in a so-called Vault. Based on these references the right secrets will be injected in a container”, Sebastiaan explains.
This offers a great advantage when it comes to security. Even if malicious users get hold of a reference, they can’t do anything with it. It is solely the reference to a secret, while the secret itself is securely stored in a Vault.
“Another advantage: developers don’t need to have access to kubectl or the Kubernetes API, unlike Sealed Secrets. In this way, you can clearly split responsibilities. Especially for a large development team, this is important. You don’t want to give every developer access to the Kubernetes API and encrypting and decrypting secrets.”
Proven solution
Vault is a proven enterprise solution, used a lot worldwide. It uses, among other, strong encryption, to always store secrets securely. “There are also extensive possibilities for configuring policies. With these policies, you determine who’s got access to which secrets. By using audit logging you can see which uses requested or edited which secrets.”
“Another important advantage are the extensive possibilities for authentication within Vault. We can give our customers access to secrets and fetching secrets from Kubernetes. This solution gives a lot of flexibility.”
Layer of isolation
“We run Vault as Software as a Service on our True infrastructure for customers. On top of this runs also a layer of automation, that looks at the namespaces of the Kubernetes cluster. This layer configures policies and determines which namespace can have access to which secrets. You can view this layer as a layer of isolation, that makes sure namespace can’t access each other’s secrets.”
In the Kubernetes clusters of True run a so-called mutating webhook. When a deployment happens with a reference to Vault, this webhook injects the desired secrets in the memory of the container.
Next to that True also provides its customer with Keycloak, an open source identity and access management solution. Keycloak gives customers access to the SaaS services True provides, like ArgoCD for GitOps, kubectl and Vault. “In this way, we provide our customer access in a secure way to their Vault. Developer can add their secrets themself. Customers can also use two-factor authentication (2FA)”, Sebastiaan explains.
Using Vault for several years
True uses Vault for several years. First within the organisation self, and for half a year now for our customers as well. “They can store their secrets themselves in Vault.” The feedback of customers is positive. “Vault lets them work easier with secrets, partly because of the intuitive interface, is their feedback. A larger development team can work directly in Vault and doesn’t need to have access to kubectl. That important, because Vault is easier to work with. Also, separation of responsibilities is an important advantage for our customers.”
Onboarding
True onboards customers in Vault. “In onboarding sessions, we show our customer the way in Kubernetes and GitOps. And, in this case, also in Vault.” In these sessions, there is attention for the challenges that arise in Vault, like debugging errors.
Sebastiaan: “We are also curious ourselves to the development of Vault and Kubernetes for interesting new possibilities. We are currently investigating how to integrate a tool like ArgoCD more specifically in Vault. This would further simplify the configuration of deployments.”