Guest post originally published on the InfraCloud blog by Nikhil Purva

As more and more organizations making the shift to cloud native technologies, Kubernetes has become the de facto choice to orchestrate container based applications. As applications grow in size, the number of microservices increases and so does the data they process. Hence, handling data, especially sensitive data becomes critical. Out of the box, Kubernetes supports ‘Secrets’ objects to store sensitive information like passwords, tokens, ssh keys, etc. securely.

Kubernetes secret eliminates the need of hard-coding sensitive data in the application code. Secrets provide this sensitive information as data mount or expose them as environment variables.

Let’s take a quick example of how this secret object is getting used within the Kubernetes cluster.

Create a secret object:

apiVersion: v1
kind: Secret
 name: rabbitmq
 namespace: default
type: Opaque

Now, let’s understand how to Inject this secret object inside a pod using an environment variable.

apiVersion: apps/v1
kind: Pod
 name: nginx
  - name: nginx
    image: nginx:latest
            name: rabbitmq
            key: RABBITMQ_PASSWORD

That’s how we can easily inject a secret inside a pod and secure sensitive information. However, there is a challenge with this approach. Our sensitive information is still not fully secure because the yaml definition of secrets are base64 encoded, and anyone can easily decode base64 encoded secrets.

To overcome this challenge with managing secrets within Kubernetes cluster, we need a better secret management system that provides a single source of credentials, secrets, security policies, etc. There are many solutions already available in the open source world like Bank-Vaults, AWS Secrets Manager, and Cloud KMS, but in this article, we will be focusing on HashiCorp Vault as it is currently widely adopted within the cloud native ecosystem. In this blog post, we will discuss how you can create a production grade secret management system using Hashicorp Vault.

What is HashiCorp Vault?

HashiCorp Vault is a secret management tool that is used to store sensitive values and access it securely. A secret can be anything, such as API encryption keys, passwords, or certificates. Vault provides encryption services and supports authentication and authorization.

We can run Vault in high-availability (HA) mode and standalone mode. The standalone mode runs a single Vault server which is less secure and less resilient that is not recommended for production grade setup.

Benefit of using HashiCorp Vault in HA using Raft

Vault can run in the multi-server mode for high availability to protect against outages. To persist the encrypted data, Vault supports many storage backends like Consul, MySQL, DynamoDB, etc.

But there are some challenges using these standard backend storages like:

So we need a solution which minimizes these challenges and at the same time, allows us to implement and manage security efficiently.

In this article we are focusing on Integrated storage to persist the data which has below benefits:

In a Kubernetes cluster, to deploy Vault in high availability mode, it would be deployed as a cluster of pods. When deploying Vault in HA mode, all the nodes in a Vault cluster will have a replicated copy of Vault’s data. Under the hood, Raft Consensus Algorithm is used to replicate the data across all the nodes.

HA Cluster diagram flow
Image Source

Deploy Vault in HA and Vault Auto Unseal

Before we jump into deploying Vault, it is important to understand what happens when we deploy a Vault pod. After deployment, the Vault pods start in a sealed state. That means until we unseal the Vault server, almost no operations are allowed. Here, unsealing is the process of decrypting the data inside Vault and allowing other services to access it.

The data that is generally stored in Vault is encrypted and it needs an encryption key to decrypt the data. As we see in the below diagram, to decrypt the data, Vault must decrypt the encryption key first, which requires the master key. Unsealing is the process of getting access to this master key. The master key is also stored alongside all other Vault data, but it is encrypted by another mechanism: the unseal key. By default the Vault config uses Shamir seals.

Diagram flow showing Deploy Vault Unsealing (shared keys -> master keys -> encrypted keys)
Image Source

Vault Auto Unseal

To make the Vault server secure, there is also an API that seals it. This process puts away the master key in memory and requires another unseal process to restore it. So in case a pod gets restarted, every time someone has to manually unseal the Vault using vault operator unseal command.

To avoid the manual step of unsealing, Vault supports automatic unsealing via cloud based key management services like Azure Key Vault, Amazon KMS, AliCloud KMS, and Google Cloud KMS which enables us to let trusted cloud providers take care of the unsealing process, as seen in the below diagram.

Diagram flow showing Vault Auto Unseal (cloud based key -> master keys -> encrypted keys)
Image Source

We are going to set up an auto-unsealing Vault on Kubernetes with Azure Key Vault. Let’s start by setting up the required Azure Services:
Prerequisite: You need to have an Azure cloud account. If not, you can create trial account for free.

Screenshot showing vault-k8s-data overview

It is very important to copy the value of Application (client ID) and client secret from register an App section mentioned above.

Screenshot showing add role assignment
Screenshot showing selections of key management operations, cryptographic operations and privileged key operations
Screenshot showing add access policy of vault-k8s-data
Screenshot showing create a key on vault-k8s-data

Once the above steps are done, it’s time to install the HashiCorp Vault. The recommended way to deploy a Vault in the Kubernetes cluster is using the Vault’s official Helm chart. To deploy Vault in HA with auto unsealing use the below-mentioned values.yml file.

  enabled: false
    repository: "hashicorp/vault"
    tag: "1.9.0"
    # Overrides the default Image Pull Policy
    pullPolicy: IfNotPresent
  # Configure the Update Strategy Type for the StatefulSet
  updateStrategyType: "OnDelete"
      memory: 256Mi
      cpu: 250m
      memory: 256Mi
      cpu: 250m
    enabled: true
    replicas: 3
      enabled: true
      config: |
        ui = true
        listener "tcp" {
          tls_disable = 1
          address = "[::]:8200"
          cluster_address = "[::]:8201"
        seal "azurekeyvault" {
          tenant_id       = "0fd16XXX-xxxx-xxxx-xxxx-xxxx"
          client_id       = "ab509eca-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
          client_secret   = "UTb7Q~xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
          vault_name      = "vault-k8s-data"
          key_name        = "vault-k8s-unsealer-key"
          subscription_id = "c83a96b1-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
        storage "raft" {
          path = "/vault/data"
        service_registration "kubernetes" {}
    enabled: true
    size: 500Mi

The values mentioned under seal “azurekeyvault” in the above configuration are derived from Azure portal while setting up azure services.


tenant_idKey Vault’s Directory ID
client_idService Principal’s Application ID
client_secretService Principal’s generated secret (the one that is not retrievable)
vault_nameName of Azure Key Vault instance
key_nameName of generated key on Azure Key Vault
subscription_idID of the Azure Subscription

Proceed with the installation following the steps mentioned below:

$ helm repo add hashicorp
"hashicorp" has been added to your repositories

$ helm install vault hashicorp/vault -f values.yaml
NAME: vault
LAST DEPLOYED: Sat Mar  5 22:14:51 2022
NAMESPACE: default
STATUS: deployed
Thank you for installing HashiCorp Vault!
Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:
Your release is named vault. To learn more about the release, try:
$ helm status vault
$ helm get manifest vault

Let’s verify the installation by running below-mentioned command:

$ kubectl get pods -n vault
NAME                                           READY   STATUS    RESTARTS 
vault-0                                        0/1     Running   0      
vault-1                                        0/1     Running   0     
vault-2                                        0/1     Running   0

As discussed earlier, Vault starts in a sealed state, and that’s why all the Vault pods are in running status but none of the Vault pods are “ready”. To unseal Vault, we need to initialize it. Just exec into pod vault-0 and initialize your Vault instance (and keep your generated keys safe) as shown below:

$ kubectl exec -it vault-0 vault operator init

Recovery Key 1: FKjt5wkzN5bUBIuR52KrPP1c2Il/f7RZdn5E+ipfNF8s
Recovery Key 2: FCzUyduESPyavh6QtqWZpdnUDKa3bEEpBHbX3NgTrCiU
Recovery Key 3: Tf7FVEpj5tdJLqQqNw/Jt0OytRI5FAZYig/yafSVz3Xg
Recovery Key 4: duLpa/6IozTOR0mkO7sp0CwmnI+1DsC6d2+oZG/A1CIZ
Recovery Key 5: pyVFs/rRFEk9rSn57Ru+KeuAQzW6eurl3j0/pS/JRpXD
Initial Root Token: s.d0LAlSnAerb4a7d6ibkfxrZy
Success! Vault is initialized
Recovery key initialized with 5 key shares and a key threshold of 3. Please
securely distribute the key shares printed above.

Now, we see the Vault unsealer in action:

$ kubectl get pods -n vault
NAME                                           READY   STATUS    RESTARTS
vault-0                                        1/1     Running   0      
vault-1                                        0/1     Running   0     
vault-2                                        0/1     Running   0         

However, our replica vault-1 and vault-2 are still not ready. They are follower pods of the leader vault-0. In order to make vault-0 visible, we need to login using our Initial Root Token.

$ kubectl exec -it vault-0 -- vault login s.d0LAlSnAerb4a7d6ibkfxrZy

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key                  Value
---                  -----
token                s.d0LAlSnAerb4a7d6ibkfxrZy
token_accessor       hJEFibLTbUP4sgA8X4tMBqZ8
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

Next, let’s join vault-1 and vault-2 to Vault-0 to make the Vault setup Highly available by running commands as shown below:

$ kubectl exec -it vault-1 -- vault operator raft join http://vault-0.vault-internal:8200

Key       Value
---       -----
Joined    true
$ kubectl exec -it vault-2 -- vault operator raft join http://vault-0.vault-internal:8200

Key       Value
---       -----
Joined    true

Let’s check the pod status and this time you would be able to see them in running as well as Ready mode:

$ kubectl get pods -n vault
NAME                                           READY   STATUS    RESTARTS   
vault-0                                        1/1     Running   0     
vault-1                                        1/1     Running   0     
vault-2                                        1/1     Running   0     

The last but important step is to verify whether the HA setup is correct or not, is by running the below command against each Vault pod and making sure that the value of HA Enabled parameters is true.

$ kubectl exec -it -n vault vault-0 -- vault status
Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  1.9.0
Storage Type             raft
Cluster Name             vault-cluster-30882e80
Cluster ID               1afbe13a-e951-482d-266b-e31693d17e20
HA Enabled               true
HA Cluster               https://vault-0.vault-internal:8201
HA Mode                  active
Active Since             2022-01-19T04:39:37.586622342Z
Raft Committed Index     61
Raft Applied Index       61
$ kubectl exec -it -n vault vault-1 -- vault status
Key                      Value
---                      -----
HA Enabled               true
HA Cluster               https://vault-0.vault-internal:8201
HA Mode                  standby
Active Node Address
Raft Committed Index     61
Raft Applied Index       61
$ kubectl exec -it -n vault vault-2 -- vault status
Key                      Value
---                      -----
HA Enabled               true
HA Cluster               https://vault-0.vault-internal:8201
HA Mode                  standby
Active Node Address
Raft Committed Index     61
Raft Applied Index       61

As you can see vault-0 is active and vault-1 and vault-2 is standby. Now, Let’s try deleting vault-0 and run the same command, you will notice that vault-1 has become active and vault-0 becomes standby.

kubectl -n vault delete pod vault-0
$ kubectl exec -it -n vault vault-1 -- vault status
Key                      Value
---                      -----
HA Enabled               true
HA Cluster               https://vault-1.vault-internal:8201
HA Mode                  active
Active Node Address
Raft Committed Index     61
Raft Applied Index       61

$ kubectl exec -it -n vault vault-0 -- vault status
Key                      Value
---                      -----
HA Enabled               true
HA Cluster               https://vault-1.vault-internal:8201
HA Mode                  standby
Active Since             2022-01-19T04:39:37.586622342Z
Raft Committed Index     61
Raft Applied Index       61

Vault Web UI

Finally to be able to access the Vault web interface; let’s port-forward the service.

$ kubectl port-forward -n vault svc/vault 8200:8200`

Vault UI will be accessible on http://localhost:8200

Screenshot showing sign in to vault page

To login in Vault, enter the token that we generated when we initialized Vault for the first time (Initial Root Token).

Screenshot showing Secrets Engines page

Secrets Engine is a Vault component that stores or generates a secret. Vault supports different types of Secrets Engines like key-value, ssh keys, certificates, etc.

To start with, let’s use a KV secrets engine, click on the Enable New Engine+ button, select the KV engine and click on Next.

Screenshot showing enable KV Secrets Engine page

Give a path name and click on Enable Engine.

Screenshot showing secret page

Now, let’s create a secret, click on Create secret and enter the details as shown in the below figure.

Screenshot showing create secret page

Once we click on save, we should be able to see that the secret has been successfully created and stored in a Vault.

Since Vault centrally stores, secures and controls access to the secrets, it is important to control the permissions before anyone can gain access. Vault supports Role Based Access Control to limit the access. Refer to the official Vault documentation on policies to understand more about the policies and roles.

Until this point, we have successfully deployed Vault in HA using Helm, learnt about configuring role based access and created a secret object. Let’s now inject this secret inside a pod.

Inject Vault secrets into pod

Since Vault has been deployed successfully let’s now jump into injecting the Vault secrets into the pods/application. Mainly there are two ways for a pod to use Secret object:

Now, we will be focusing on injecting a Vault secret into an application using External Secrets. The benefit of Kubernetes External Secrets is that it can be used with Vault, external secret management systems like AWS Secret Manager, Azure Key Vault to securely add secrets. Kubernetes External Secrets extends the Kubernetes API by adding an ExternalSecrets object using Custom Resource Definition.

An ExternalSecret object declares how to fetch the secret data, while the controller converts all ExternalSecrets to Secrets.

Kubernetes cluster towards HashiCorp Vault diagram flow

Image Source

Just like the Hashicorp Vault, the recommended way to deploy Kubernetes-External-Secrets is using the official helm chart and use the below-mentioned values.yaml file.

  VAULT_ADDR: http://vault:8200
  WATCHED_NAMESPACES: ""  # Comma separated list of namespaces, empty or unset means ALL namespaces.
  LOG_LEVEL: info
  # Specifies whether RBAC resources should be created
  create: true

  # Specifies whether a service account should be created
  create: true
  # Specifies annotations for this service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: kubernetes-external-secrets
replicaCount: 1
  tag: 8.5.1
  pullPolicy: IfNotPresent
resources: {}

Run the following command to deploy the external secrets.

$ helm repo add external-secrets
"external-secrets" has been added to your repositories

$ helm install k8s-external-secrets external-secrets/kubernetes-external-secrets -f values.yaml
NAME: k8s-external-secrets
LAST DEPLOYED: Wed Mar 23 22:50:35 2022
NAMESPACE: default
STATUS: deployed
The kubernetes external secrets has been installed. Check its status by running:
$ kubectl --namespace default get pods -l ","
Visit for instructions on how to use kubernetes external secrets

At this point, the kubernetes-external-secret has been deployed successfully and with the help of the ExternalSecrets object, now we can inject secrets inside the pods that are stored in Vault.

But the external-secrets controller needs to be authenticated with Vault before making any request to retrieve the secrets, so we need to enable the Kubernetes authentication method and attach a policy to it.

Enable Kubernetes Auth Method

Let’s access the Vault pod’s terminal and enable kubernetes auth method:

$ kubectl exec -it vault-0 sh
$ vault login s.d0LAlSnAerb4a7d6ibkfxrZy
$ vault auth enable kubernetes

Success! Enabled kubernetes auth method at: kubernetes/

The Kubernetes Auth Method can be used to authenticate with Vault using Kubernetes Service Account Token, as Vault accepts this token by any client within the Kubernetes cluster.

Configure the kubernetes authentication method by running below command from inside the active Vault pod:

$ vault write auth/kubernetes/config \
    token_reviewer_jwt="$(cat /var/run/secrets/" \
    kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \

Success! Data written to: auth/kubernetes/config

The token_reviewer_jwt and kubernetes_ca_cert are created for a pod by Kubernetes when it is started. The env variable KUBERNETES_PORT_443_TCP_ADDR is defined and references the internal Kubernetes API Server network address. When an application tries to authenticate with Vault using its Service Account Token, Vault uses the above configuration to verify the client application’s identity with Kubernetes API Server.

Attach a policy to Kubernetes auth method

Since we have already created a readonly policy, let’s create a role for kubernetes and attach this policy. In the below mentioned role all the service accounts in any namespaces are allowed to authenticate. We can also limit authentication to specific service accounts and namespaces as well.

Diagram flow showing attach a policy to Kubernetes auth method
$ vault write auth/kubernetes/role/k8s-role \
    bound_service_account_names=* \
    bound_service_account_namespaces=* \
    policies=readonly \

Success! Data written to: auth/kubernetes/role/k8s-role

Let’s now inject vault secrets inside a pod. We will be using the same example which we used at the start of this blog post. As discussed earlier, we just need to create an ExternalSecret object and the controller will convert it to a Secret object. Delete the secret object we created earlier and apply the below file.

apiVersion: ''
kind: ExternalSecret
 name: rabbitmq
 namespace: default
 kvVersion: 1
 backendType: vault
 vaultMountPoint: kubernetes
 vaultRole: k8s-role
   key: secret/Rabbitmq
   property: username

Save this file by name rabbitmq-external-secret.yaml and run:

$ kubectl delete secret rabbitmq
secret "rabbitmq" deleted

$ kubectl apply -f rabbitmq-external-secret.yaml 

Let’s run the below command to check the status of external-secret.

$ kubectl get
rabbitmq   10s         SUCCESS   15s

$ kubectl get secrets
NAME       TYPE     DATA    AGE
rabbitmq   Opaque   1       15s

So here, External Secret has been created successfully and the External Secrets fetched the correct value from Vault, and created the Secret object for us. This Secret object is used by our application as we saw in the beginning of the post.


Since Kubernetes is widely used and to make a Kubernetes cluster production-grade, it is very important to make sure the secret objects are handled carefully. In this article, we learned why it is important not to use native secret objects due to their limitations and the importance of using a secret management tool like HashiCorp Vault.

We also learned about External Secrets, which help us to inject secrets inside a pod. With the help of external secrets, we can easily replace the Kubernetes secrets with Vault secrets inside any existing deployments.

That’s it, folks !! I hope this article was informative and engaging. I am looking forward to hearing your thoughts on this post, so let’s connect and start a conversation on LinkedIn.

Looking for help with cloud native security? do check out our capabilities how we’re helping startups & enterprises as an DevSecOps consulting services provider.