Guest post by Saylor Berman, Senior Software Engineer, F5 NGINX and Ciara Stacke, Senior Software Engineer, F5 NGINX
Using minimal Docker containers is a popular strategy in the world of containerization due to benefits of security and resource efficiency. These containers are stripped-down versions of traditional containers, designed to contain just the essential components needed to run an application. In some cases, a base container may contain nothing at all (this would be the `scratch` container).
Many people approach the creation of Docker images by making it as simple as possible to get their app working. This involves choosing a base image such as `ubuntu` or `python` that contains all the necessary libraries and tooling to get up and running. While easy, these images have an increased attack surface and memory footprint due to all the extra “stuff” inside.
When we developed NGINX Gateway Fabric, our implementation of the new Kubernetes Gateway API, we leveraged several tools and strategies to ensure security and optimization. In this article we cover our design decisions:
- Reduced Surface
- Image Security Scanning
- Code Quality and Security
- Deployment Security Best Practices
Reduced Surface
We chose to start as small as possible. In fact, we can’t get any smaller. Based on `scratch`, our `nginx- gateway-fabric` image simply contains the Golang binary and nothing else. There are a couple steps taken before we produce the final build, but we ensure that the final build only contains what is necessary, which is just the binary. The Dockerfile looks something like this:
```Dockerfile
FROM alpine:3.18
RUN apk add --no-cache libcap
COPY ./build/out/gateway /usr/bin/
RUN setcap 'cap_kill=+ep' /usr/bin/gateway
FROM scratch
USER 102:1001
ENTRYPOINT [ "/usr/bin/gateway" ]
```This multi-step Dockerfile allows us to use `alpine` with the latest version of libcap to copy our pre-built binary into the proper location and set the necessary permissions for us to manage nginx. We then use `scratch` as the base for our production image and set the user and group ID so that we can control and limit permissions that our container has access to.
With this approach, the `nginx-gateway-fabric` image is roughly the size of the binary itself, without any extra bloat. The binary does not need any extra dependencies to run and we have kept the size and attack surface as small as possible.
Image Security Scanning
One of the most effective ways to keep your product secure is by doing regular security scanning. With the help of Trivy, we run regular image security scans as part of our Github CI/CD pipeline. Trivy scans the container image for any known vulnerabilities (CVEs) that exist in libraries or binary files. Using the `scratch` image protects us from any vulnerabilities in the base image, but Trivy can still catch any vulnerabilities in the libraries that are compiled in our Golang binary. Results of the scan are uploaded to the
Github Security tab for our repository to make it easy for our team to see any issues that are found.
Code Quality and Security
In addition to minimizing our container images, we also prioritize code quality and security within our application, employing a security-first mindset. We prioritize security from the outset, thoroughly evaluating each design and feature with a focus on security. We proactively identify and safeguard assets at the early stages of our processes, ensuring their protection throughout the development lifecycle, and we adhere to best practices for secure design, including proper authentication, authorization, and encryption mechanisms. An important part in upholding these standards is our use of a robust code review process. Every contributor to the project must open a pull request with any changes, and each pull request must be approved by at least two project maintainers. In addition to this process, we also incorporate the use of linters and tools to maintain our high standards for code security and quality.
Static Code Analysis
One way to boost code quality is to use Static Code Analysis tools (commonly abbreviated as SAST). One of the key advantages of using SAST is the ability to detect vulnerabilities early in the development process. We integrate CodeQL, a static code analysis tool developed by GitHub, into our CI/CD pipeline, which scans and identifies any potential issues in the pull request, as well as in the main branch. This allows us to remediate any potential code quality or security issues before we merge a change to the main branch. Similar to the Trivy scanning, any potential issues are uploaded to the GitHub Security tab for our repository to make it easy for our team to see any issues that are found.
Keeping Dependencies Up-to-Date
Keeping your project’s dependencies up-to-date is crucial for security and performance. In the NGINX Gateway Fabric project, we utilize Dependabot to automate this process by continuously monitoring our project’s dependencies and automatically opening pull requests when updates are available. This ensures that we’re always using the latest, most secure versions of our dependencies.
Dependabot also provides security alerts by monitoring the Common Vulnerabilities and Exposures (CVE) database. When a security vulnerability is identified in one of our project’s dependencies, Dependabot promptly notifies us and automatically opens a pull request with the necessary updates. This, along with scanning our Docker images, enables us to patch vulnerabilities quickly.
Deployment Security Best Practices
As a critical component in a Kubernetes environment, we strive to follow Kubernetes security best practices wherever possible. This includes:
- Least privilege — We designed our RBAC (Role Based Access Control) specification to have the minimum required permissions for the resources NGINX Gateway Fabric needs to access. Only permissions explicitly required for our normal operation are defined.
- Principle of Single Responsibility — Each containerized application should have a single responsibility, meaning it should perform only one task or function. Following this principle, we designed NGINX Gateway Fabric as a multi-container application, with the binary running in a separate container from the NGINX binary.
- Hardened Container Security Context — We have configured the Security Context on each container to include the following
- We follow “minimal Linux capabilities” model by dropping all capabilities from the containers running in the NGINX Gateway Fabric Pod, and explicitly add only those that are required for the project to operate. This helps to mitigate the risk of potential privilege escalation attacks on the containers.
- We ensure that no process running in the containers can gain more privileges than its parent process. This is achieved by setting the `AllowPrivilegeEscalation` flag to false in the containers’ security context.
- Our containers are configured with a read-only root filesystem. A read-only root filesystem helps to enforce an immutable infrastructure strategy. To achieve this, we configure the containers to write to mounted volumes, and set the `readOnlyRootFilesystem` flag to true in the containers’ security context.
- We set the `runAsUser` and `runAsGroup` flags to non-root values. Additionally, we set the `runAsNonRoot` flag to true in the Pod Security Context to enforce that the containers must run as non-root users.
What’s next?
Hopefully these secure development practices have given you some ideas on how to minimize risk of exposure while also developing resilient apps and choosing platform tools. We’d love to hear if you’re interested in using NGINX Gateway Fabric to improve security posture of your Kubernetes platform. We encourage you to:
- Join the project as a contributor
- Try the implementation in your lab
- Test and provide feedback