Project post by Joel Hans
Today, Argo is four independent projects that help organizations leverage GitOps for running enormously complex Kubernetes workflows—hundreds of steps, thousands of nodes, and hundreds of thousands of concurrent processes.
Like this screenshot of an Argo workflow, which Hong Wang, founder and CEO of Akuity, shared with me after an interview with the core Argo team:
Every single dot is an Argo workflow step, each running in a separate container within this user’s Kubernetes cluster and representing one part of their CI/CD pipeline. We can’t decipher what they’re running, but we do know they’re using Argo to automate and orchestrate an extraordinarily complex cloud native application.
On one hand, Wang agonizes over the anonymity of this user—he’d like to know who runs this workflow and where—but on the other, he’s just proud when the Argo community steps up.
He says, “When the community is willing to showcase what they’re doing, that makes me feel like I’m building something amazing.”
And Argo has proven to be enormously valuable for not just CI/CD pipelines, but also machine learning, infrastructure automation, and processing vast volumes of data—like CERN and the results from experiments in the Large Hadron Collider. But before we talk about all the fantastic recent results from the Argo team, I need to step back to 2017 and give you a peek into the decisions that set Argo up for open source success.
The road to Argo 2.0… and beyond
More than five years ago, the team at Applatix needed a new tool for managing their infrastructure: a framework for designing and orchestrating parallel jobs in Kubernetes. The team included not just Wang, but also Jesse Suen, the co-founder and CTO of Akuity, and Ed Lee, the founder of Applatix.
Argo moved quickly. In August 2017, the Applatix team launched Argo (the project that now goes by Argo Workflows) on GitHub. In October of the same year, they pushed the re-implementation of Argo’s workflow engine as a Kubernetes CRD (Custom Resource Definition), further expanding the community.
In January 2018, just months after Applatix launched Argo, they were acquired by Intuit, which wanted to connect more with developers and practitioners doing GitOps and application lifecycle management. They struggled around running Kubernetes clusters at scale, and there was no better team to solve those internal problems—then reflect them to the open source community—than the Argo founders.
Much has changed for Argo and its team in the final months of 2017, and even more so in the years since, but the founding team firmly believes in the decisions they made—and lessons they learned—from starting an open source project on the right footing.
Lessons learned from Argo’s toddler years
Are you public from that first commit? Are you working in stealth in a private GitHub repository to build an MVP of your project before going public? Are you building an internal tool that you happen to share with others? Are you building for a community that’s wider than your own needs?
These are questions Argo’s founding team wrestled with in their early days. That you will, too, if you’re starting an open source project, and your answers will inevitably ripple through how your project and who—as vendors and end users—for years to come.
Let’s look at what the Argo team feels like they got right from the get-go—and what they learned along the way.
Don’t start an open source project from a siloed state
Wang firmly believes in the team’s decision to start from open source from the very first day and very first commit. The first commit to the argo-workflows repository is the project’s birth, and he believes that decision prevented their team from starting in a siloed environment, where their only priority is developing a tool that’s only useful to their core team.
That decision became all the more relevant when Applatix was acquired by Intuit, which wanted to use Argo in production. Had Argo started from a siloed state at Applatix, the acquisition could have quickly sent them down a dangerous road where they only prioritized solving Intuit’s unique problems around scaling Kubernetes. Useful for their internal processes but entirely out of the conversation with industry trends, standards, or users.
Wang says, “Open source forces you to connect with the latest technology and the latest best practices. That inversely affects the technical decisions to make your overall platform better.”
Remember that there’s a wide and meaningful margin between a project that happens to be open source and a project that’s designed for an open source community from day one.
Consider use cases beyond your own
As the Argo team peered out well beyond a siloed state, they searched for ways to make their project valuable far beyond the offices of Intuit.
Jesse Suen, co-founder and CTO of Akuity: “Without open source, and if you’re only building for your organization, you have a narrow focus. Open source forces you to consider the use cases of other organization’s needs, and it helps shape it to be an overall better product at the end of the day.”
Ed Lee, the founder of Applatix, agrees, saying their decision to stick with open source kept them honest, keeping them from building niche products that had no relevance to the broader cloud native ecosystem.
They collaborated early with potential vendors and end users, resulting in significant contributions from organizations like Red Hat and Blackrock. Suen relates the “constant drumbeat of people” asking the Argo team for more ways to manage many applications.
Intuit didn’t need a new ApplicationSet feature, but they formulated the specifications and use cases such a feature needed to solve. They passed the ball to a tiger team at Red Hat, which owned the repository and its initial development, and eventually merged the feature as a first-class component of Argo CD—not because Intuit needed it, or the Argo team prioritized it, but because they nurtured use cases most important to their community.
Suen says, “I think that’s just a testament of open source working really well.”
Stick to standards
Argo’s team argues that to build an open source project with longevity, you need to stay apprised of the trends and best practices of the community you’re trying to serve. That’s how you encourage contributions from developers and practitioners eager to contribute to the wider benefit of their community, not a single siloed project.
Your project needs to combine with others into large toolchains for developing or deploying cloud native applications. That’s how you encourage adoption from end users that need to quickly adapt their current practices around your tooling without rebuilding from scratch.
The standards-based approach pays dividends today at Argo. Wang says, “Argo is becoming not only a product people use directly, but also a foundational engine for other open-source products. That’s a good recognition for our efforts. People want to build things on top of Argo, and you can only build on top of a good foundation.”
One powerful example is CERN, which uses Argo CD and Crossplane to orchestrate massive infrastructure across every cloud. Dan Garfield, co-founder of Codefresh and a member of the Argo core team, says, “They use Argo CD to orchestrate the provisioning of all the infrastructure to run their workflows, and then can instantly spin up the entire power of the planet to do all the data crunching for black holes. It’s mind-blowing. And then can shut it all down on a dime. I was speechless for 40 minutes.”
That’s only possible now because, years ago, Argo committed to standards-based, cloud-agnostic tooling.
Look for grassroots community support first, not corporate sponsors
Today’s financial reality means that any open source project needs investment. Argo was no different, with support for the core team first coming from Applatix and Intuit. They could have nurtured relationships with potential vendors, hoping to find another company with financial incentive to pay toward Argo’s future, but instead they focused on solving real-world problems for a huge variety of end users.
Lee says, “I don’t think you could have a vibrant, multi-vendor product like Argo without end-user participation. Without them, it would be a project that’s primarily sponsored or dominated by one vendor. End-user involvement in the community is essential.”
If you ask the Argo team, a hundred active, engaged, and passionate community members are likely worth more to the longevity of your project than a single large-scale sponsor. A community-driven approach reassures newcomers that your project is well-governed, not favoring a single vendor or operating on their whims. And it’s validation that you’re invested in the project for the long-haul, not liable to disappear on a whim.
Peering into Argo’s open source future
Today, Argo thinks of itself as a close cousin to Kubernetes in the CNCF Cloud Native Landscape: They’re supported by multiple vendors, with heavy end-user contributions, and have high code velocity and contributor diversity. They’re both building a high-quality, high-value product with longevity.
I would argue that the data backs up their claim, with 370,000+ contributions and nearly 2,000 contributing companies—an increase of 124% since Argo joined CNCF.
For more information on the data behind Argo’s contributor diversity, development velocity, and community health, see our Project Journey Report.