Today’s post is written by Javier Salmeron, Engineer at Bitnami
Many experienced Kubernetes users may remember the Kubernetes 1.6 release, where the Role-Based Access Control (RBAC) authorizer was promoted to beta. This provided an alternative authentication mechanism to the already existing, but difficult to manage and understand, Attribute-Based Access Control (ABAC) authorizer. While everyone welcomed this feature with excitement, it also created innumerable frustrated users. StackOverflow and Github were rife with issues involving RBAC restrictions because most of the docs or examples did not take RBAC into account (although now they do). One paradigmatic case is that of Helm: now simply executing “helm init + helm install” did not work. Suddenly, we needed to add “strange” elements like ServiceAccounts or RoleBindings prior to deploying a WordPress or Redis chart.
Leaving these “unsatisfactory first contacts” aside, no one can deny the enormous step that RBAC meant for seeing Kubernetes as a production-ready platform. Since most of us have played with Kubernetes with full administrator privileges, we understand that in a real environment we need to:
- Have multiple users with different properties, establishing a proper authentication mechanism.
- Have full control over which operations each user or group of users can execute.
- Have full control over which operations each process inside a pod can execute.
- Limit the visibility of certain resources of namespaces.
In this sense, RBAC is a key element for providing all these essential features. In this post, we will quickly go through the basics (for more details, check the video below) and dive a bit deeper into some of the most confusing topics.https://www.youtube.com/embed/CnHTCTP8d48
The key to understanding RBAC in Kubernetes
In order to fully grasp the idea of RBAC, we must understand that three elements are involved:
- Subjects: The set of users and processes that want to access the Kubernetes API.
- Resources: The set of Kubernetes API Objects available in the cluster. Examples include Pods, Deployments, Services, Nodes, and PersistentVolumes, among others.
- Verbs: The set of operations that can be executed to the resources above. Different verbs are available (examples: get, watch, create, delete, etc.), but ultimately all of them are Create, Read, Update or Delete (CRUD) operations.
With these three elements in mind, the key idea of RBAC is the following:
We want to connect subjects, API resources, and operations. In other words, we want to specify, given a user, which operations can be executed over a set of resources.
Understanding RBAC API objects
So, if we think about connecting these three types of entities, we can understand the different RBAC API Objects available in Kubernetes.
- Roles: Will connect API Resources and Verbs. These can be reused for different subjects. These are binded to one namespace (we cannot use wildcards to represent more than one, but we can deploy the same role object in different namespaces). If we want the role to be applied cluster-wide, the equivalent object is called ClusterRoles.
- RoleBinding: Will connect the remaining entity-subjects. Given a role, which already binds API Objects and verbs, we will establish which subjects can use it. For the cluster-level, non-namespaced equivalent, there are ClusterRoleBindings.
| TIP: Watch the video for a more detailed explanation.
In the example below, we are granting the user jsalmeron the ability to read, list and create pods inside the namespace test. This means that jsalmeron will be able to execute these commands:
But not these:
Example yaml files:
Another interesting point would be the following: now that the user can create pods, can we limit how many? In order to do so, other objects, not directly related to the RBAC specification, allow configuring the amount of resources: ResourceQuota and LimitRanges. It is worth checking them out for configuring such a vital aspect of the cluster.
Subjects: Users and… ServiceAccounts?
One topic that many Kubernetes users struggle with is the concept of subjects, but more specifically the difference between regular users and ServiceAccounts. In theory it looks simple:
- Users: These are global, and meant for humans or processes living outside the cluster.
- ServiceAccounts: These are namespaced and meant for intra-cluster processes running inside pods.
Both have in common that they want to authenticate against the API in order to perform a set of operations over a set of resources (remember the previous section), and their domains seem to be clearly defined. They can also belong to what is known as groups, so a RoleBinding can bind more than one subject (but ServiceAccounts can only belong to the “system:serviceaccounts” group). However, the key difference is a cause of several headaches: users do not have an associated Kubernetes API Object. That means that while this operation exists:
this one doesn’t:
This has a vital consequence: if the cluster will not store any information about users, then, the administrator will need to manage identities outside the cluster. There are different ways to do so: TLS certificates, tokens, and OAuth2, among others.
In addition, we would need to create kubectl contexts so we could access the cluster with these new credentials. In order to create the credential files, we could use the kubectl config commands (which do not require any access to the Kubernetes API, so they could be executed by any user). Watch the video above to see a complete example of user creation with TLS certificates.
RBAC in Deployments: A use case
We have seen an example where we establish what a given user can do inside the cluster. However, what about deployments that require access to the Kubernetes API? We’ll see a use case to better understand this.
Let’s go for a common infrastructure application: RabbitMQ. We will use the Bitnami RabbitMQ Helm chart (in the official helm/charts repository), which uses the bitnami/rabbitmq container. This container bundles a Kubernetes plugin responsible for detecting other members of the RabbitMQ cluster. As a consequence, the process inside the container requires accessing the Kubernetes API, and so we require to configure a ServiceAccount with the proper RBAC privileges.
When it comes to ServiceAccounts, follow this essential good practice:
Have ServiceAccounts per deployment with the minimum set of privileges to work.
For the case of applications that require access to the Kubernetes API, you may be tempted to have a type of “privileged ServiceAccount” that could do almost anything in the cluster. While this may seem easier, this could pose a security threat down the line, as unwanted operations could occur. Watch video above to see the example of Tiller, and the consequences of having ServiceAccounts with too many privileges.
In addition, different deployments will have different needs in terms of API access, so it makes sense to have different ServiceAccounts for each deployment.
With that in mind, let’s check what the proper RBAC configuration for our RabbitMQ deployment should be.
From the plugin documentation page and its source code, we can see that it queries the Kubernetes API for the list of Endpoints. This is used for discovering the rest of the peer of the RabbitMQ cluster. Therefore, what the Bitnami RabbitMQ chart creates is:
A ServiceAccount for the RabbitMQ pods.
A Role (we assume that the whole RabbitMQ cluster will be deployed in a single namespace) that allows the “get” verb for the resource Endpoint.
A RoleBinding that connects the ServiceAccount and the role.
The diagram shows how we enabled the processes running in the RabbitMQ pods to perform “get” operations over Endpoint objects. This is the minimum set of operations it requires to work. So, at the same time, we are ensuring that the deployed chart is secure and will not perform unwanted actions inside the Kubernetes cluster.
Final thoughts
In order to work with Kubernetes in production, RBAC policies are not optional. These can’t be seen as only a set of Kubernetes API Objects that administrators must know. Indeed, application developers will need them to deploy secure applications and to fully exploit the potential that the Kubernetes API offers to their cloud-native applications. For more information on RBAC, check the following links: