By AccuKnox October 26, 2023

Member post by AccuKnox

Introduction:

In the realm of cybersecurity, ensuring that virtualized or cloud-based infrastructures security is paramount. One crucial aspect is safeguarding applications where most of our crown-jewel sits and are susceptible to dynamic changes. In modern infrastructure Kubernetes deployments are gaining popularity by over 55% annual rate (ref). And hence, securing Kubernetes deployment (applications) and going one-step further to protecting the underlying host system becomes critical. This technical blog explores the enhanced security the user can get by combined use of AWS Bottlerocket and KubeArmor (a runtime security engine). Both are powerful tools that work in harmony to provide robust runtime protection for Kubernetes environments end-to-end. AWS Bottlerocket is originally developed and currently maintained by Amazon Web Services (AWS) and is well-optimized for use with Amazon Elastic Kubernetes Service (EKS), although it can also be used with other container orchestration platforms like Kubernetes.

What is AWS Bottlerocket?

AWS Bottlerocket: A Security-Focused Linux OS Distribution which is tailored for securely running containerized workloads in the cloud environment. It was developed with a view to have  a security-focused, open-source Linux-based operating system which will be specifically designed for running any type of containerized workloads in all kinds of cloud environments. AWS Bottlerocket does not allow installing any package management tools or maintenance binaries on the host system. Users can only utilize these tools through a “control pod” or an admin port, ensuring strict control over their underlying os infrastructure.

Rocket illustration

What is the significance of AWS Bottlerocket?

Enhanced Security: One of the primary reasons for the significance of Bottlerocket is its strong emphasis on security. It is purpose-built to provide a secure and minimalistic environment for running containers. By reducing the attack surface and removing unnecessary components, Bottlerocket helps mitigate potential vulnerabilities and enhances the overall security posture of containerized workloads.

Bottlerocket can be used with  –

Secure Updates with The Update Framework (TUF)

Another essential aspect of AWS Bottlerocket’s security is the implementation of secure updates for the operating system image and root file system. This is achieved through The Update Framework (TUF), a CNCF graduated project designed for secure system updates. Leveraging TUF ensures that updates are tamper-proof and come from trusted sources.

Locking Down with Kernel Integrity Features

To prevent unauthorized access and tampering, AWS Bottlerocket uses a kernel feature called “integrity,” which prohibits processes from writing to the kernel memory or loading/unloading kernel modules. Additionally, the distribution utilizes “dm-verity” (device mapper verity), a kernel module feature that offers integrity verification of the root file system through cryptographic digests. Any attempt to write to sensitive file paths in the root file system is denied and reported.

What is KubeArmor?

KubeArmor is a runtime security engine which is currently a CNCF sandbox project. The aim of KubeArmor is to implement Zero Trust Security by auto discovering application behavior and auto generating granular policies based on the observed app behavior. These policies are usually enforced as guardrails using Linux Security Modules (LSMs) to prevent any unknown binary execution, or application behavior

Leverage the unfair KubeArmor Advantage

KubeArmor at its core support observability of the application behavior via eBPF and enforces granular policies at the kernel-level for in-line mitigation of those zero day attacks in a proactive way. 

As Zero Trust security calls for ‘never trust always verify” and ‘assume breach’ requirements its imperative to have multi-layer defense for the modern workload. Here is how KubeArmor helps its customer to reach their Zero Trust journey

Problem Statement:

A Remote Code Execution and other attack vectors were detected within an already ‘secured’ cluster and code repositories. Customer X helps its users to detect the delta difference the PR (pull request) will be impacting the main repository and highlight the security implications of the GitHub action. 

Diagram flow showing process from Dev User to K8s Cluster by Atlas Module

Upon any trigger such as PR, Customer X Atlas module (depicted below) invokes a pod that has 2 containers –

The functionality of Init-dependencies container is to clone the target user’s repo and its associated dependencies locally and then exit. These repo and dependencies are downloaded into a volume mount point.

This volume mount point location is then assessed by Analyzer container

Point of concern:

After the dependencies are downloaded, there are events where there is a “postscript” that could be invoked and an attacker could leverage this “postscript” to launch an potential (RCE) attack.

Their requirements to mitigate such attacks were well articulated and thought-upon which are as follows –

Solution:


While AWS Bottlerocket safeguards the host and worker nodes, KubeArmor steps in to protect the Kubernetes pods and their applications. Many applications installed by vendors or users lack sufficient security understanding, making them vulnerable. KubeArmor minimizes the attack surface of such pods by allowing only specific processes to execute, access certain sensitive file paths, and use specific network primitives.

Diagram flow showing solution overview

KubeArmor secures Kubernetes service account tokens and secrets by tightly controlling access to these critical resources. The security policies can restrict access to a specific set of processes, ensuring only authorized entities can interact with sensitive data.

Diagram flow showing process of securing Kubernetes secrets and service account tokens

KubeArmor supports the implementation of least permissive policies, following the tenets of zero-trust security. This approach grants explicit access to a predetermined set of binaries and operations while denying all other activities, thus minimizing potential vulnerabilities.

Diagram flow showing KubeArmor applying least permissive policies

Leveraging eBPF for Enhanced Visibility and Observability

KubeArmor utilizes eBPF (extended Berkeley Packet Filter) to enhance observability and visibility. eBPF is a powerful technique that provides real-time telemetry and alerting, replacing older mechanisms like kernel audit. This ensures comprehensive monitoring without compromising system performance.

Diagram flow showing KubeArmor leveraging eBPF for enhanced visibility and observability

Demonstration: Combining AWS Bottlerocket and KubeArmor

To demonstrate the synergy between AWS Bottlerocket and KubeArmor, we present a hands-on demo within an EKS cluster with Bottlerocket nodes.

Diagram flow demonstration of combining AWS Bottlerocket and KubeArmor

Scenario 1: Blocking Access to Kubernetes Service Account Tokens

In this scenario, we use KubeArmor to implement a policy that denies access to the Kubernetes service account token. This token, mounted within a predefined volume bound point, is vital for interacting with the Kubernetes API server. The policy effectively blocks any unauthorized access to this critical token.

Scenario 2: Denying Execution of Certain Processes

KubeArmor is employed to prevent the execution of certain processes like “apt” and “apt-get” within pods. These package management tools are unnecessary and can increase the attack surface, making them unfit for production environments. The policy restricts their execution, minimizing potential risks.

Scenario 3: Enabling Specific Network Primitives

In this final scenario, we showcase how KubeArmor allows specific processes, like Apache and Ping, to access certain network primitives (UDP, TCP, and RAW). This enables fine-grained control over network operations while denying access to other protocols, enhancing security.

Conclusion:

By combining AWS Bottlerocket and KubeArmor, Kubernetes deployments can achieve robust runtime protection. While Bottlerocket ensures a secure foundation for the host and worker nodes, KubeArmor provides granular control over pods and applications, enhancing overall security in the cloud-based ecosystem. The powerful features of both tools complement each other, creating a formidable defense against cybersecurity threats within Kubernetes environments. KubeArmor provides enhanced security by using BPF-LSM to protect Bottlerocket containers from within by limiting system behavior with respect to processes, files, etc. For e.g., a k8s security access token that is mounted within the pod is accessible by default across all the containers. KubeArmor can restrict access to such tokens only for certain processes. Similarly KubeArmor can be used to protect other sensitive information e.g., k8s secrets, x509 certs, within the container.