KubeCon + CloudNativeCon + Open Source Summit China Virtual sponsor guest post from Eric Adams, Cloud Software Engineer at Intel, and Sakari Poussa, Cloud Solutions Architect at Intel
You clicked on this article puzzling, “Isn’t the service mesh already encrypted and secure?” You had configured “mtls: STRICT” and per the documentation mutual TLS communication is the default. Further, you followed all Kubernetes security best practices, including setting up different namespaces, securing the host, and adopting a trustworthy cloud provider. What can go wrong? The privacy of your entire service mesh depends upon the secrecy of its private key, the key used to sign the certificates used in mutual TLS. If leaked, an intruder could snoop in on, decrypt, observe, and monitor YOUR service mesh. That would be terrible.
How can someone gain access to your service mesh private key? Typically, that private key is generated by the service mesh as a self-signed key and is stored as a base64-encoded Kubernetes secret. Anyone with kubectl admin access can just read it with a simple “kubectl get secret my-private-key -o json” and then do a base64 decode. Kubernetes does have support for using a key management service (kms) to manage the key, which avoids having to store it in a secret. That helps protect from the rogue Kubernetes admin. However, you will have to setup and manage a kms yourself or “trust” the vendors’ kms. If the kms runs as a local service to your cluster, then a very clever hacker with host access could theoretically read the unencrypted memory to get to the key.
What if there was a way to securely store your service mesh private key without having to “trust” someone’s kms? What if it was easy to use and didn’t require rewriting software or rebuilding your container images? Hardware-based memory encryption technologies provide CPU enforcements and attestation to assert that it is genuine. It enables software applications to protect sensitive parts of their running code and data in encrypted and cryptographically isolated enclaves. Even someone with admin access to the host system and/or cluster with the ability to install tools to snoop network traffic or read memory will only be able to see encrypted information.
How does it work?
The key part of the solution is the SGX Operator that implements with the k8s Certificate Signing Request APIs, which matured to V1 in Kubernetes release v 1.22. The operator creates an enclave, loads the key signing code, and attests the enclave. Once established, the CA key is created in the enclave and all future certificate signing happens within the enclave, with no visibility from the outside. All of this is done transparently to the user. You do not have to read any SGX manual or re-write any code. The SGX operator does all the hard work and is as easy to install as “kubectl apply -f sgx-operator.yaml”. (Please note that this code will be open sourced and is expected to become available at the end of December 2021.) Further, Istio makes it easy to configure an external Certificate Authority (CA), such as cert-manager or the above SGX Operator, to handle all certificate signing.
You do, however, need a processor with Intel® SGX support. Your Kubernetes cluster infrastructure provider handles backend details such as attestation and leveraging the SGX device plugin (which controls the number of enclaves and the containers that may use them per node). Below we describe each of the steps in greater detail.
- When the SGX operator pod first starts up the enclave is attested. The key generation and signing code are initialized into it. The private key is then created within the enclave, which is protected by encryption.
- Configure your service mesh to use an external CA for key signing. This is done via setting the K8S_SIGNER parameter, set it to use the SGX operator. Once started, the public key is copied into the secret named external-ca-cert.
- Once istiod is running, start a pod with auto injection set to enabled. This will ensure that each pod will automatically start a side car container to serve as proxy, which could be envoy or any other istio proxy. The proxy will generate its own key pair and then request istiod to sign the certificate with the service mesh private key.
- The above certificate signing request is then passed to the SGX Operator to sign, which holds the signing key inside its enclave, never exposing it at any time. The signed certificate is then returned through istiod to the proxy, enabling it to start communicating securely on the service mesh.
- All pods, regardless of language, will go through the same process to sign their self-generated certificates. The service mesh leverages the private key stored in the encrypted enclave for signing all certificates, making your service mesh more secure.
The video at: https://networkbuilders.intel.com/intel-software-guard-extensions-intel-sgx-securing-private-keys-in-an-encrypted-enclave-for-your-service-mesh-demo demonstrates the same. We have plans to similarly enhance the Kubernetes cert-manager.
In conclusion, the SGX Operator supports confidential compute, encrypting the service mesh private key while in use and at rest, and in so doing, makes your service mesh communications more secure.
Eric Adams is a Cloud Software Engineer at Intel with 20 years of experience focused on advanced cloud use cases, securing workloads, and using hardware accelerators.
Sakari Poussa is a Cloud Solutions Architect at Intel with nearly 25 years of experience. He is currently focused on open-source service mesh technologies, working on securing and accelerating them.