Posted on October 31, 2022

Guest post originally published on Flux’s blog

Software supply chain attacks are one of the most critical risks threatening today’s software and have begun to collapse like a dark cloud over the software industry. For the Flux family of projects we are taking precautions against these threats. Apart from implementing security features and best practices, it is important to us to educate our users. You can find all Flux’s security articles here. Today we will talk about a new security feature.

Internet meme showing developer trying to reach software supply chain attacks yet being pulled by Flux

Let’s start with a brief historical explanation of how we got to this point. It all started with the following sentence:

Flux should be able to distribute and reconcile Kubernetes configuration packaged as OCI artifacts.

RFC-0003: Flux OCI support for Kubernetes manifests.

From then on, the Flux community worked hard and brought this feature with Flux v0.32. So with that, you can store and distribute various sources such as Kubernetes manifests, Kustomize overlays, and Terraform modules as OCI (Open Container Initiative) artifacts with Flux CLI and tell Flux to reconcile your sources that are stored in OCI Artifacts, and Flux will do that for you. 🕺🏻

But this only covered the first stage of the entire implementation. There is more than that. ☝️

One of the most exciting features of this RFC is the verification of artifacts. But why, what is it, is it really necessary or just a hype thing? This is a long topic that we need to discuss. Suppose you store the cluster desired state as OCI artifacts in a container registry. How can you be one hundred percent sure that the resources that Flux reconciles are the same as the resources that you’ve pushed to the OCI registry? This is where the verification of artifacts comes into play. But, how can we do that? 🤔

Thanks to the Sigstore community we have a great set of services and tools for signing and verifying authenticity. One of the tools is cosign which can be used for container signing, verification, and storage in an OCI registry. We will use it to verify the authenticity of the OCI Artifacts in Flux. Starting with v0.35, Flux comes with support for verifying OCI artifacts signed with Sigstore Cosign. Documentation for setting it up can be found here.

Let’s jump right into the details of how we can actually use it.

We will deploy cert-manager by storing its manifests in OCI registry packaged as an OCI Artifacts, using the Flux CLI. Then we are going to sign it with cosign and configure Flux to verify the artifacts’ signatures before they are downloaded and reconciled.

You need three things to complete this demo;

Let’s start by creating a simple Kubernetes cluster:

kind create cluster

Let’s install Flux on it – if you need to use other options, check out the installation page.

export GITHUB_USER=developer-guy
export GITHUB_TOKEN=$GITHUB_TOKEN

flux bootstrap github \
  --owner=$GITHUB_USER \
  --repository=flux-cosign-demo \
  --branch=main \
  --path=./clusters/my-cluster \
  --personal

⚠️ Note: Don’t forget to change the values with your own details!

First we download the cert-manager install manifests from GitHub:

curl -sSLO https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml

https://github.com/cert-manager/cert-manager/releases/tag/v1.9.1

Next we push the manifests to GitHub container registry with Flux CLI:

mkdir -p ./manifests
 cp cert-manager.yaml ./manifests
$ flux push artifact oci://ghcr.io/$GITHUB_USER/manifests/cert-manager:v1.9.1 \
  --path="./manifests" \
  --source="https://github.com/cert-manager/cert-manager.git" \
  --revision="v1.9.1/4486c01f726f17d2790a8a563ae6bc6e98465505"
► pushing artifact to ghcr.io/developer-guy/manifests/cert-manager:v1.9.1
✔ artifact successfully pushed to ghcr.io/developer-guy/manifests/cert-manager@sha256:d1fb0442865148a4e9b4c3431c71d8e44af56c3eb658ea495c5ec48d48c6638b

Before signing the OCI artifact with Cosign, we need to create a set of key pairs, a public and private one:

cosign generate-key-pair

This command above outputs two files to disk: cosign.pub and cosign.key. The cosign.pub file is the public key and the cosign.key file is the private key. You can use the cosign.pub file to verify the container image and the cosign.key file to sign the container image.

To let Flux to verify the signature of the OCI artifact, we should create a secret that contains the public key::

kubectl -n flux-system create secret generic cosign-pub \
  --from-file=cosign.pub=cosign.pub

Now, let’s sign it:

$ cosign sign --key cosign.key ghcr.io/$GITHUB_USER/manifests/cert-manager:v1.9.1
Enter password for private key:
Pushing signature to: ghcr.io/developer-guy/manifests/cert-manager

As we stick into the GitOps practices, we should create a file that contains the OCIRepository resource, then commit and push those changes into the upstream repository that Flux watches for changes:

git clone git@github.com:developer-guy/flux-cosign-demo.git
cd flux-cosign-demo

Let’s create a secret with the GitHub token:

$ flux create secret oci ghcr-auth \
  --url=ghcr.io \
  --username=${GITHUB_USER} \
  --password=${GITHUB_TOKEN}
► oci secret 'ghcr-auth' created in 'flux-system' namespace

Configure Flux to pull the cert-manager artifact, verify its signature and apply its contents:

cat << EOF | tee ./clusters/my-cluster/cert-manager-sync.yaml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: cert-manager
  namespace: flux-system
spec:
  interval: 5m
  url: oci://ghcr.io/${GITHUB_USER}/manifests/cert-manager
  ref:
    semver: "*"
  secretRef:
    name: ghcr-auth
  verify:
    provider: cosign
    secretRef:
      name: cosign-pub
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: cert-manager
  namespace: flux-system
spec:
  interval: 1h
  timeout: 5m
  sourceRef:
    kind: OCIRepository
    name: cert-manager
  path: "."
  prune: true
  wait: true
EOF

Let’s commit and push these changes:

git add .
git commit -m"Add cert-manager OCIRepository and Kustomization"
git push

After couple of seconds for Flux will have applied these changes. Now let’s check the status of them:

Or, you can trigger the reconcilation immediately by running the simple command: flux reconcile source git flux-system

For Kustomization resources:

$ flux get kustomizations
NAME            REVISION                                                                SUSPENDED       READY   MESSAGE                                        
cert-manager    v1.9.1/d1fb0442865148a4e9b4c3431c71d8e44af56c3eb658ea495c5ec48d48c6638b False           True    Applied revision: v1.9.1/d1fb0442865148a4e9b4c3431c71d8e44af56c3eb658ea495c5ec48d48c6638b
flux-system     main/14f1e66                                                            False           True    Applied revision: main/14f1e66

For OCIRepository resources:

$ flux get sources oci
NAME            REVISION                                                                SUSPENDED       READY   MESSAGE                                        
cert-manager    v1.9.1/d1fb0442865148a4e9b4c3431c71d8e44af56c3eb658ea495c5ec48d48c6638b False           True    stored artifact for digest 'v1.9.1/d1fb0442865148a4e9b4c3431c71d8e44af56c3eb658ea495c5ec48d48c6638b'

If you see the status of the OCIRepository is True, it means that Flux has successfully verified the signature of the container image. Because Flux adds a condition with the following attributes to the OCIRepository’s .status.conditions:

If the verification fails, Flux will set the SourceVerified status to False and will not fetch the artifact contents from the registry. If you see the status of the Kustomization is True, it means that Flux has successfully applied the manifests that are stored in the container image.

Let’s check the status of the cert-manager deployment:

$ kubectl get pods --namespace cert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-cainjector-857ff8f7cb-l469h   1/1     Running   0          76s
cert-manager-d58554549-9fbgj               1/1     Running   0          76s
cert-manager-webhook-76fdf7c485-9v82g      1/1     Running   0          76s

Furthermore

As we can store Helm Charts in OCI registries with the release of Helm v3.8.0 which means that we can also sign them with cosign and verify them with Flux. The Flux community is already working on it and want to add support for verifying the Helm charts stored in OCI registries as OCI Artifacts in the next releases of Flux. You can follow the progress of this feature in the following issue: fluxcd/source-controller#914.

The Sigstore community is aware of the risks and the toil of managing public/private key pairs, so cosign offers another mode for signing and verification called Keyless, which do not require managing any keys manually. Flux also supports that. If you omit the .verify.secretRef field, Flux will try to verify the signature using the Keyless mode. It’s worth mentioning keyless verification is an experimental feature, using custom root CAs or self-hosted Rekor instances are currently not supported.