By Michael Crenshaw July 19, 2022

Project post cross-posted from the Argo Blog by Michael Crenshaw

In early 2022, the Argo team and CNCF began work with Ada Logics to perform a security audit on the four Argo projects. Ada Logics discovered a number of issues which were then either triaged or resolved by the Argo team. This post summarizes the findings and notes a few things learned along the way.

We are very grateful to CNCF for helping drive this effort, to OSTIF for financing the audit, and to Ada Logics for actually conducting the audit.

Summary of findings

Ada Logics identified twenty-six issues: seven in Argo CD, six in Argo Workflows, and thirteen in Argo Events. Seven CVEs were issued for Argo CD, and two CVEs were issued for Argo Events. The remaining issues are being tracked as non-CVEs.

Argo CD

Four of the seven issues identified by Ada Logics were new vulnerabilities. One of the seven (#5 below) was an old vulnerability from a previous audit (it had not been considered a vulnerability at the time, but our thinking has since shifted — see notes below about multi-tenancy). Another vulnerability from the previous audit (#2 below) had not been addressed.

One additional vulnerability (#3) was discovered by the Argo team while reviewing notes from the Ada Logics draft report.

All seven vulnerabilities have been patched. They are listed below in descending order of severity. Where applicable, the Ada Logics issue number (ADA-ARGO-SA-*) and/or the issue number from the previous audit, conducted by Trail of Bits, (TOB-ARGO-*) is included.

  1. External URLs for Deployments can include javascript (CVSS 9.0, Critical) (ADA-ARGO-SA-02)
  2. Insecure entropy in PKCE/Oauth2/OIDC params (CVSS 8.3, High) (TOB-ARGO-017)
  3. Certificate verification is skipped for connections to OIDC providers (CVSS 8.3, High)
  4. Denial of service through large directory app manifest files (CVSS 6.5, Medium) (ADA-ARGO-SA-17)
  5. Symlink following allows leaking out-of-bounds YAML files from Argo CD repo-server (CVSS 4.3, Medium) (ADA-ARGO-SA-15)
  6. Symlink following allows leaking out-of-bound manifests and JSON files from Argo CD repo-server (CVSS 4.3, Medium) (TOB-ARGO-018/ADA-ARGO-SA-16)
  7. A leaked API server encryption key can allow XSS for SSO users (CVSS 2.6, Low) (ADA-ARGO-SA-09)

All issues which were not evaluated to be vulnerabilities are being tracked in a GitHub project.

Patterns

Path traversal and symlink following in the repo-server

Preventing path traversal and out-of-bounds symlink following in the Argo CD repo-server is difficult, due to the many places where paths and symlinks are read during manifest generation. There have been three vulnerabilities involving path traversal and three vulnerabilities involving symlinks (all of moderate or high severity).

Argo CD 2.4 went a long way towards mitigating these attacks. Each repository in the repo-server’s on-disk cache now gets its own randomized directory, and permissions are locked down when manifest generation is not actively in progress. And plugins no longer share the same cache directory as the built-in config management tools (Helm, Kustomize, etc.).

Even with these improvements, defeating symlink attacks remains a priority. Argo CD 2.5 will completely block out-of-bounds symlinks in all repositories by default, whether or not those symlinks appear to be involved in manifest generation.

Until stronger unprivileged filesystem protections (user namespaces) gain wider support with cloud providers, path traversal and symlink-following risks will require constant vigilance. In the meantime, the Argo CD team will continue to evaluate other mitigations, such as per-tenant repo-server segmentation.

Evolving operator security needs

With more Argo CD operators using single instances for multiple tenants, the project’s way of evaluating security concerns has shifted. For example, we used to consider source files from git as trusted. But due to the possibility of an adversarial tenant exploiting path traversal, we now must treat source files as potentially-malicious.

The rise of multi-tenancy has also affected how we calculate vulnerability severity scores. For example, when considering the impact of a cross-site scripting attack via CVE-2022–31035, we evaluated the worst case scenario: a malicious tenant hijacking an admin account to attack other tenants. That analysis yielded a 9.0 (critical) score. Scoring this way helps us communicate urgency to the most security-sensitive operators.

Processes

The Argo CD team is extremely proud of recent work to improve how we handle each reported vulnerability. The issues reported by Ada Logics gave us an opportunity to test and tweak our processes.

The security team now tracks vulnerability reports as issues in a private GitHub repository. The issue template includes a checklist to make sure reports are handled uniformly and that important steps are not missed.

Once we’ve established that a vulnerability is valid, we create a GitHub Security Advisory (GHSA) with a private fork to develop a patch. We also write a detailed Security Advisory with a standard set of components such as affected versions and mitigations.

Standardized issue tracking in GitHub also gives us an easy way to look back and evaluate what we’ve done and identify trends. For example, since we started using the GitHub issues, we have taken a median of 39 days from initial report to release patches (min. 20, max. 70). If this number shifts over time, we can evaluate what we are doing right (or wrong) to move the needle.

Community

As the core Argo CD team has worked on vulnerabilities reported by Ada Logics, we’ve been delighted by community-led activity to contribute security enhancements such as deny rules in Project destinations and symlink blocking. This work demonstrates the vitality of the Argo community, and will pay huge dividends by mitigating or eliminating future vulnerabilities. Thank you, Argo community, you are awesome!

Argo Events

Among the audit findings by Ada Logics for Argo Events, there are 2 high severity vulnerabilities that should be particularly mentioned.

One of them is about the potential path traversal and symbol link attack attack when using Git as the triggering source artifactory. As mentioned in the Argo CD portion, this is a pattern across multiple Argo projects. We have made an effort to mitigate this issue in version v1.7.1+.

The other high severity issue revealed was about using a deprecated package ioutil without enforcing read limit on I/O operations, which could potentially cause DoS attack for all the webhook based EventSources such as BitBucket, Gitlab, etc.

Besides these, another issue revealed by the audit was the use of deprecated or unmaintained libraries, which could potentially mean including undiscovered and not reported vulnerabilities. We have been making efforts to move away from those libraries. We recommend upgrading Argo Event to version v1.7.1+ to make it more secure.

Summary

The Argo team owes a huge thanks to the CNCF, OSTIF, and Ada Logics for sponsoring, facilitating, and conducting this security audit. If you have questions about the report, drop a comment here or visit us in CNCF Slack.

Links