Member post originally published on the CyberArk blog by John Walsh
Handling secrets in cloud-native environments is a challenge for many organizations. Virtually any application requires some sort of secret, such as a database password, a service token, or a certificate, or to establish secure connections. The growing popularity of Kubernetes in the last few years means that many security administrators are now searching for the capabilities to properly handle secrets in Kubernetes services. However, built-in secret capabilities of Kubernetes are limited: they lack certain functions that are essential for securely handling secrets.
Managing secrets insecurely poses many risks. Secrets leaked in source code, in application logs, or in another way, may fall into the wrong hands, with potentially disastrous results.
How Kubernetes handles secrets
Kubernetes provides a Secret object to securely store secret values. There are two types of secrets in Kubernetes:
- Built-in secrets – Kubernetes Service Accounts automatically generate credentials that are used to securely access the Kubernetes API.
- Custom secrets – You can create a Secret object yourself to store sensitive data.
Kubernetes secrets can be created in one of the following ways:
- Automatically – Existing files with sensitive data can be stored as Secret objects with the kubectl command:
kubectl create secret my-secret –from-file=./credentials.txt
- Manually – You can author a Secret manifest in either JSON or YAML:
apiVersion: v1 kind: Secret metadata: name: my-secret data: username: bXktdXNlcm5hbWUK password: bXktcGFzc3dvcmQK type: Opaque
This object is then created based on the manifest using kubectl apply.
Limitations of Kubernetes secrets
There are several limitations to the method Kubernetes uses for storing secrets.
First, as the sensitive values in the example manifest above are base64 encoded – and thus not encrypted – it is easy to decode these values with the following command:echo bXktdXNlcm5hbWUK | base64 –decode my-username
Therefore, the secret’s manifest cannot be stored beside any other manifests (Deployments, Services, and so on) in Git. You don’t want to store a plain-text sensitive value in your repository!
Second, Kubernetes uses etcd as the storage for the secrets. Etcd is a distributed key/value storage known for its great performance. However, it lacks some features that are vital for sensitive data, such as an audit log, insights into key age, and automatic key rotation.
In addition, by default, Kubernetes stores secrets in etcd unencrypted, which means that anyone with access to the etcd cluster has access to the secrets.
Third, anyone with root access to the Kubernetes nodes can access all secrets by impersonating kubelet. Kubernetes currently shares all secrets with all nodes. Preventing privilege escalation is, therefore, essential for blocking access to secrets.
Fourth, Kubernetes can make secrets available to pods either through exposing them as environment variables or by mounting them as files containing plain-text secrets. Environment variables can easily expose sensitive values through, for example, debug logs. Plain-text files on disk are available to everyone with access to the pod.
Finally, Kubernetes’ RBAC functionality provides only get and set permissions for secrets. When you get a secret, you can only get its decrypted value. A more secure zero-trust setup would allow a developer to set a secret and then only retrieve the encrypted value for consumption. This is not available natively in Kubernetes.
Protecting secrets in container environments
A common approach to getting more secure secret management on Kubernetes is to introduce an external secret management solution, such as Hashicorp Vault, AWS Secrets Manager, Azure Key Vault, or Google Secret Manager. While this approach resolves some issues inherent in Kubernetes, other – more general – secret management challenges persist. With a third-party solution, it is still virtually impossible to ensure that:
- Secrets are available only to those containers that actually need them. Tight integration with Kubernetes is required to map secrets to containers and to get proper insight into this.
- Secrets are not stored on disk or in environment variables.
- Applications cannot leak secrets.
- Inter-system secret transfer is always TLS-encrypted.
- Secrets are regularly, automatically rotated
In addition, many third-party solutions don’t solve the Secret Zero problem.
Kubernetes secrets and CyberArk Conjur
The best way to deal with a secret is to… not deal with it at all. CyberArk Conjur can take care of it for you.
Conjur uses a sidecar container called Secretless Broker to proxy authenticated requests to external systems such as databases. The broker authenticates with CyberArk Conjur natively, using multi-factor authentication, to get access to the required secrets. This essentially resolves the Secret Zero issue. Secret management is handled completely separately from the application, which eliminates the risk of leaking the secret. The application simply has no access to the secret!
The Secretless Broker never stores secrets on disk, and uses strong encryption in all communication with CyberArk Conjur, as well as in connections to the protected external systems.