Guest post originally published on InfraCloud’s blog by Vishal Biyani
In the decade of 2000, a lot was happening in the software industry. The Toyota lean movement had proven to be of significant value in the production of cars. A lot of industries beyond manufacturing and obviously the software industry took note of the parallels to “lean” in their world. The early innovators in software industry also were dabbling with the idea of “continuous integration” as can be seen from birth of tools such as CruiseControl (2001), and Hudson (2005) – a fork of which was named Jenkins (2011) was adopted by the OSS community & became one of the most popular tools to implement CI/CD! In year 2009, the official term DevOps was coined at O’Reilly Velocity Conference where Flickr engineers presented their famous talk: “10+ Deploys per Day: Dev and Ops Cooperation at Flickr” & DevOpsDays conference was formed.
(From the popular talk “10+ Deploys per Day: Dev and Ops Cooperation at Flickr”)
The basic idea of DevOps was to align the “Development” and “Operations” teams towards the goal of delivering software at higher speeds. While the alignment of technologies, processes & people took a good part of the decade of 2010!
Cloud, scale & complexity growth: decade of 2010
The decade of 2010 saw tremendous growth in cloud, SaaS & technologies like containers. Just to get a sense of the growth, AWS went from 100K customers in 2010 to a million in 2015 or Docker which was introduced in 2013 had 100K companies using/evaluating in 2014, and within a year 2015, a million companies were using containers in the form of Docker!
(Source: Docker Adoption Research by Datadog)
The growth of cloud & technologies did change a few things such as:
- A whole set of new tools were born to build, operate and scale software. For example Terraform – a way to define infrastructure in HCL. Or once Docker became mainstream – the orchestration became the new ground for innovation & eventually led to Kubernetes and a very large ecosystem around it named Cloud Native Technologies.
Growth in orchestrators such as Mesos, Docker Swarm, Kubernetes or Rancher Cattle in early years. Source: Docker Adoption Research by Datadog 2017
- SaaS – Many new businesses were now delivered in form of SaaS over the internet. This meant you could now have a large portion of your application needs delivered by a SaaS provider while you could focus on the core of your business. In that process the concerns of operations were not just about deploying & managing infrastructure. The operations now included building for higher reliability, or building with security as a focus because your services are crossing the network perimeters of the organization!
As SaaS became more mainstream – things like reliability, security, cost all took the main stage. The scope of “operations” expanded way more than before!
ShiftLeft + “You Build It, You Run It” (YBYRI)
While the scope of operations expanded way more, the things on Dev side were also increasing equally. A lot of concerns around things such as security or reliability were being defined “as code”. Hence, developers had to be aware of these things, even if at a superficial level. Shifting left was also driven by the agile movement so that fixes can happen as close to the developer’s inner development as possible instead of the code going all the way to stage for the required changes and flowing through the entire pipeline again.
Some large organizations also experimented with YBYRI – You Build It, You Run It! Which meant the developer teams had to be aware of many things beyond just writing code. The developer teams could end up owning evetything from changes in code related to business to reliability, security etc.
Development+++: definitions & confusion!
So DevOps movement started with great intent and did solve many problems. However, the growth of cloud, SaaS & complexity of applications blurred the boundaries quite fast – leading to increased cognitive load for almost everyone!
So should a developer apart from writing code also know how Terraform provisions everything or how to write all manifests and charts for Kubernetes? Similarly, should operations team end up writing code and if yes, where do you draw the boundaries? It is not a question of creating walls, but definitely the cognitive load of so many things on everyone is not productive. So if we go back to our original diagram of Dev aligning to Ops, it is now an alignment between many more functions such as security, reliability, cost, and of course, Dev & Ops!
Rise of Platform Engineering
All these things lead to re-thinking of the structure of teams to reduce cognitive load, increase flow state & faster feedback loops. Some organizations which were scaling fast & were leading adopters of DevOps were quick to recognize the pattern and improvise the team structures, tools & processes to build out platforms which enable developers.
So what is platform engineering? The goal of platform engineering within an organization is to enable the delivery teams to get what they need & move fast while focusing on application. With the rise in complexity & breadth of things that an application goes through, it is impossible for developers to know everything in depth. This is where the platform enables them to use sensible defaults for things they want to move fast on, while later changing it to their needs. Over time the platform will become a full blown product that enables the delivery of software but that does not have to be the goal to start with.
That’s a wrap! You can learn more about Platform Engineering in our detailed blog post on Platform Engineering 101. If you’re looking to get started with platform engineering, do learn about Backstage and the need for IDPs in the development ecosystem and how it can help organizations to grow efficiently. For a weekly dose of cloud native technologies, don’t miss to subscribe to our newsletter.