Guest post by Dylen Turnbull, F5 NGINX Solution Architect
GitOps as a sandwich line for developers
Following on the heels of DevOps and Platform Ops, we now have GitOps – a new stylish way to set up and provision infrastructure. While it’s easy to write off GitOps as just the latest flavor-of-the-month, I see it as a potential sea change and signal moment in how we build the technology stacks that run modern applications. Why? GitOps is the moment we cross the Rubicon and move fully into the realm of 100% declarative – infrastructure-as-code the way we all hoped it would be. By that, I mean declarative all up and down the stack, even at the application layer – without manual processes but with continual drift detection.
With the adoption of proper GitOps, you can actually “shift left” with a fully declarative process for defining resources and a fully automated process for deploying, modifying, maintaining, and monitoring them to keep systems running in the desired state without the need to manually check for dependencies and remediate any issues.
A GitOps Primer
For the uninitiated, GitOps is a methodology for managing infrastructure and applications that uses Git as the central repository for instructions and the single source of truth. In GitOps, users declaratively define an application and its underlying infrastructure in Git. The deployment and management of the application and infrastructure is largely automated, enabling simple versioning and rollbacks.
It’s worth emphasizing that GitOps – unlike other infrastructure-as-code methodologies – is by definition 100% declarative. If an approach is not 100% declarative, it’s not GitOps. Another key distinction is that the GitOps tooling constantly watches the state of the application and infrastructure, as well as checking for any new code or configuration changes. Combining deployment and monitoring into a single interface and application programming interface (API) radically reduces complexity, improves efficiency, and allows Development and Operations to truly coalesce in the same continuous integration/continuous deployment (CI/CD) pipeline-driven environment. It allows developers to define what their apps need through a common API while helping Ops teams to manage the more onerous tasks associated with operational standards, auditing and security.
Up to now, infrastructure-as-code has been at worst a mixed bag of semi-manual processes that were mostly opinionated and imperative and at best a partially declarative hybrid beast that required constant care and feeding from all interested parties. With GitOps, we can finally move freely into the world where custom resource definitions (CRDs) and APIs drive provisioning – or in other words, where a curated set of options are coded into simple YAML manifests with enough choices to satisfy even the most particular of developers but without the need for DevOps watchers to make sure nothing runs aground.
In a sense, GitOps is like the sandwich-building counter at Subway® – an assembly line for infrastructure and code with numerous options built atop CI/CD pipelines designed for smooth assembly down to even relatively bespoke combinations of technology choices. You pick the bread, the meat (prewrapped in handy 2.5-ounce packages), and the condiments. The sandwich moves down the line to the vegetable station before dressings or pepper get sprinkled on and then, finally, you can decide if you want it toasted or not. Within that assembly line are a million combinations, but all accessible via the standard menu.
McDonald’s, Subway, and a Recipe for Tasty GitOps
The utility of a Subway-like setup is why, I believe, Kubernetes has been such a runaway success – it offers a fully declarative management structure mapped to a manifest/menu with a common mechanism for delivering choice CRDs all called via a common set of APIs.
In contrast, lots of organizations are trying to give developers GitOps but it’s often not true GitOps and lacks secret sauce. In other words, they are trying to foist a McDonald’s model on developers when what they really want and need is a Subway one!
To understand what I mean, let’s look at these two famous fast-food chains in more depth. McDonald’s is known as the leader in technology adoption in the restaurant business. They are laser-focused on product consistency. Initially, McDonald’s believed it did not need to cook anything to order. Food was cooked and placed in hot bins where it could be easily grabbed and tossed into carry-out bags or onto customer trays. That worked until Burger King came along and offered cooked-to-order food and the slogan “Have it your way!”.
Customers wanted it their way, it turns out, and in reaction McDonald’s decided to cook some items to order. Even so, compared to Subway both Burger King and McDonald’s offer limited choices of meat, filling, and condiments. And both rely on a heavy infrastructure of customized hardware and software.
Subway, in contrast, is automated but low-fi at the same time, in a way that enables not only efficiency but also flexibility and portability. You can run a Subway-style sandwich line in your house or on a picnic table in the park for your kid’s birthday party.
So how does this map onto our definitive GitOps recipe? To start with, we want a declarative engine for establishing a neutral control plane across all environments – cloud, hybrid, hosted, and serverless. This is essential for GitOps because it allows users to declaratively define the desired state of cloud resources in Git and then automatically provision and manage those resources in a consistent way across different cloud providers. An example of this is Crossplane, the CNCF open source project, but there will certainly be more solutions like it as GitOps takes hold.
Next comes the deployment layer. A great example of GitOps-friendly CD is Argo. Unlike most CI/CD tooling, which is focused on the front-end of the deployment process, Argo not only automates workflows but also monitors what’s happening with applications and makes adjustments to keep them running to spec. It does this through progressive delivery and event-based dependency management. In this approach, changes to one resource automatically propagate to other dependent resources, signaled by events. This approach allows developers and Ops teams to automatically keep a system “on recipe” in the desired state. The system reacts to events automatically rather than asking beleaguered Ops teams to check dependencies.
The beauty of these newer tools is they are not resource intensive. You do not need a separate Kubernetes management cluster provisioned solely to manage your deployment processes. To translate to real-world terms, this makes bootstrapping GitOps relatively straightforward:
- Set up your developer tools.
- Build your developer cluster and locally include the tools.
- Use the developer cluster to bootstrap a permanent management cluster.
- Transfer ownership to make the management cluster the progenitor of all the production clusters spun up using recipes you define.
Until very recently, declarative recipe-driven deployment had weak, if any, support at the application layer. In sandwich-building terms, you got bread and meat but you couldn’t make declarative, recipe-driven choices for veggies and condiments. Recognizing this gap, some smart developers created an open source project called the Open Application Model (OAM). The OAM specification creates universal application-design building blocks, much like Crossplane Composite Resources.
OAM standards make platforms like KubeVela possible. With these standards, you can declare and check into Git a recipe for your choice of frontend and backend. Then you can declare a recipe for your database needs and check that into Git.
When application developers want to build something new, they can go to the repo with predefined manifests, pull in choices for a MongoDB database, a Node.js backend, and a web frontend, then check them all into the new project’s Git repo being monitored with Argo. From that recipe, the developers can alter any customizable elements to support their application their way.
The result is a recipe that can be handed around but also slightly altered by other users as necessary: if someone wants stone-ground mustard instead of yellow mustard, the architecture supports that. If more choices are needed, just hit up the Ops team responsible for the care and feeding of the CRD and voila, you can add new stuff that plays by all the rules set by a specific Ops team, without having to scramble to build a completely new recipe in a hurry.
The Legendarium Cookbook: One Recipe to Rule Them All
This talk of recipes got me thinking about – believe or not – the fantasy world created by J.R.R. Tolkien in his Lord of the Rings trilogy. I mentioned earlier that much of the power of GitOps emerges from Git’s role as the central repository of declarative instructions for building apps – the single source of truth, something like Tolkien’s “one Ring to rule them all”.
In Tolkien’s legendarium, a collection of myths about the prehistory of the world where the Lord of the Rings story takes place, the stalwart reader learns that the Ring is not itself the real source of power. The real power is in the process or “recipe” for creating the rings, which Sauron the deceiver under the guise of Annatar (“Lord of Gifts”) used to manipulate the elven smith Celebrimbor into forging the Rings of Power. It was this same “recipe” that allowed Sauron to forge the One Ring to rule them all. Considering this it could be inferred there was actually, one recipe to rule them all.
This offers us another perspective on GitOps. The recipe and its wielder is what gives GitOps its power, not the specific moving parts. As you implement GitOps, you need to avoid vendor lock-in at all costs. Even though GitOps is a sea change in app development and deployment, that doesn’t mean it’s the ultimate solution. Software methodologies will continue to evolve and you will need to update your GitOps recipe to fit a new internal development process.
For now, the open source tooling around GitOps is fascinating and you can use it to mount your Subway assembly line in any environment and truly make it your own. You can even hand it off to the developers and let them run the sandwich line. This finally makes “shift left” what it was always intended to be – a menu with near infinite choice but minimal toil built on proven recipes and systems. Kubeway your way, and tasty!