Guest post from Richard Collins, Jetstack
In recent months the cert-manager user community has been surveyed to understand how this CNCF Sandbox project is being used in production environments. As a highly popular developer-centric tool used primarily to secure Ingress endpoints with TLS encryption, cert-manager is built to give developer teams effortless machine identity management when deploying certificates to Kubernetes clusters. cert-manager removes the manual toil associated with the need to provision an X.509 certificate for each workload and will automatically renew and manage the lifecycle of each certificate.
The survey results, now compiled and released by Jetstack a Venafi company provide very insightful reading. Multi-cloud and hybrid-cloud infrastructure is now the norm for many organisations using cert-manager. Also identified are a variety of deployment scenarios where cert-manager is now used in production for service mesh and more generally securing private “pod-to-pod” traffic. However, amongst the findings, it seems security teams are largely unaware of the vital role cert-manager plays in the security set-up of each new production cluster and this can lead to security exposures which this blog highlights.
Self-hosted Kubernetes with hybrid clouds is increasingly popular.
When looking to understand more about the Cloud Service Providers (CSPs) being used to support Kubernetes production environments with cert-manager, the survey asked users to indicate which CSP is the primary provider of their production infrastructure. The key finding is that while there is an interesting distribution of CSPs, self-hosted Kubernetes with hybrid clouds appears to be increasingly prevalent. If including multi-cloud infrastructure, more than a third of respondents reported they do not use any particular public cloud as their primary provider.
Clearly many organisations feel confident in running Kubernetes on self-hosted infrastructure, as opposed to relying primarily on a managed service from a CSP. Given cert-manager is designed to be cloud agnostic, it perhaps not surprising to
see it used so prominently across multiple cloud providers and self-hosted infrastructure.
86% of new production clusters are created with cert-manager deployed as standard practice to manage the issuance and renewal of TLS and mTLS certificates.
Machine identity management using cert-manager is vital for production environments that are scaling and is clearly used as policy for bootstrapping each new Kubernetes cluster. Consistency in the way new clusters are created and avoiding “snowflake” clusters so each production cluster can be supported uniformly is important to ensure efficient cluster operations which will reduce the overall toil for SRE teams.
Not properly maintaining cert-manager in production is a security risk.
47% of users say that production clusters are not running the latest released version of cert-manager. So this is signalling a problem. For most organisations, the number of clusters is growing so it is vital to ensure all security tooling in production clusters is running the latest version. Running inconsistent versions of the same tool across multiple clusters can result in “security drift” and will imply potential future toil and cost for SRE teams.
cert-manager is increasingly used across a range of different certificate use cases.
Although unsurprisingly, 95% of users reported Ingress TLS using as the primary use case for cert-manager in production clusters, using cert-manager for service mesh solutions is becoming more important. 19% of respondents reported cert-manager is used for service mesh in production. Furthermore, many users are clearly extending their Kubernetes environments and using cert-manager to support webhook admission controllers. Overall, the data points to a more extensive use of cert-manager in production environments suggesting it is being used to meet a variety of needs.
Certificate misconfigurations are a major security concern when using Kubernetes.
Security risks from all types of misconfigurations inside production clusters are of general concern. Yet the danger of not having specific monitoring of Ingress endpoints, and of not being able to remediate certificate misconfigurations stand out as key areas of potential risk amongst users.
Properly managing these misconfigurtations are key to improving the security posture and reducing the attack surface from bad actors who can exploit and specifically target these vulnerabilities. Users of cert-manager are clearly concerned about misconfigurations that can manifest as a result of the extremely high volume of certificates that are sometimes needed when using Kubernetes. In particular, there is strong general concern around not having full visibility of the configuration status of all certificates in production environments.
Perhaps worryingly, 57% indicated manually signed certificates continue to be used in their current production infrastructure. Manually signed certificates for Kubernetes workloads will not scale and are not compatible with modern zero-trust security needs.
If you would like your own copy of the findings from Jetstack, you can access an infographic here.