Guest post originally published on Ozone’s blog

Preface

Everyone talks of DevOps and how their organizations are going about transforming their development and operational processes. Investments are high and measuring the real-world outcomes of these DevOps transformations is quintessential. Towards this, we have the DORA metrics. However, in this article, we highlight why this is inadequate and where the gap lies.

While DORA metrics focus on technical KPIs, how does an organization measure the long term business impact? For example, if you automate your testing and shift-left security scans or invest in securing your releases, there is no doubt time and effort saved while improving on quality. But has this really impacted on the long term?

The need to measure DevOps performance

You’ve heard the term “DevOps transformation” a number of times but ever wondered how it plays a role in your business achievements and why should you leverage it?

A DevOps transformation requires a significant amount of time, money and capital to an organization, necessitating it to gauge whether its resources and efforts are taking shape in the right direction to give the expected business results.

But realistically, every team requires different resources for different needs. Value stream mapping here helps to a great extent. In that, it helps increase visibility throughout the CI/CD stages, hence making it easier to visualize and map value to be delivered across from code to customers.  A Value stream delivery platform (VSDP) saves time, skills and reduces the costs by half! One such VSDP is OZONE that not only has these features but strongly focuses on enterprise value, providing higher quality to their customers with impeccable performance.

Coming back to the topic at hand, let us see what exactly is DevOps transformation and the areas where DORA metrics fall short of.

DevOps Transformation

Transforming DevOps means bringing in lean thinking and automation together with latest tech, practices, and tools leading to better business outcomes.

The main DevOps goal is to optimize the flow of the value from idea to end-user. Most software and IT Operations teams have adopted DevOps practices to evolve faster. Still, there are companies that are under pressure to adopt it in order to bid/compete with the rival companies, innovate faster and deliver better digital experiences.

Typical challenges faced by DevOps adopters:

The list of challenges is a mix of various issues, but they can all be overcome by focusing on people, processes, and technology. The boundaries between the business, development, and operations have to be removed by developing a robust DevOps transformation roadmap. However, when it comes to adopting the DevOps philosophy, the most critical factor to all the organizations remains the same, i.e., quantifying the long-term benefits of DevOps. Yet, this still seems to be overlooked and is lost in the process.

While teams focus on DORA metrics, it is still not sufficient to capture these long-term benefits of DevOps. Let us first look at what these metrics are followed by how the shortfalls can be tackled.

DORA (DevOps Research and Assessment) metrics help make performance measurement easy and make it available for teams to consume it and take actions wherever needed.

DORA’s four eyes that help you measure DevOps performance:

DORA has set the DevOps standards using which the organizations can identify the metrics for Elite, High, Medium, and Low performing teams. According to this, Elite teams are twice as likely to meet or exceed their organizational performance goals. A detailed visual below from the 2021 State of DevOps report:

Table showing software delivery platform metrics for Elite, High, Medium and Low performing teams
Software delivery platform metrics

Why is DORA not sufficient to show the big picture?

DORA metrics have become a key indicator to check the progress:  It is easy to fetch data for these metrics and it is often believed to be the sole indicator. But let’s not forget that high-performing organizations also look at customer insights and delivered values even more so than these metrics. .

Moreover, relying on one set of metrics for various products and teams is challenging as every organization is different. As there are structural differences between you and other organizations when it comes to products, team size, adaptability and experience of the team.

DevOps is a change of mindset rather than of technology. Hence, according to Gartner’s report, through 2022, 75% of DevOps initiatives will fail to meet expectations due to organizational learning and change issues.

Because of this, dealing with mere metrics is not done. Organizational learning and team dynamics may not show their impact on DORA metrics but may very well lead to better business outcomes, which contradicts DevOps measurement!

Hence, DORA metric alone cannot show you the big picture or the long term benefits of DevOps transformation.

Let us try moving ahead of DORA and look at the DevOps maturity assessment of an organization and see if this helps show the long-term impacts.

How is DevOps Maturity assessed?

DevOps Maturity is considered a model that dictates an organization’s position in the DevOps journey. It defines the current status and what needs to be accomplished to achieve the desired results. DevOps maturity is measured by keeping the following pointers in mind;

A perfect DevOps maturity model operates on three levels:

To be able to determine the maturity levels, the DevOps must comprise three components:

5 Stages in DevOps Maturity Journey

Diagram flow showing the 5 stages in DevOps Maturity Journey

While the stages give you a complete picture of DevOps status in the organization, this maturity model comes with other benefits too!

With the DevOps maturity model you can improve organizational workflow, enhance release frequency, and make production faster.

Other metrics that help to quantify long term ROI of DevOps transformation

Checkout some of the crucial areas of analysis that leaders can use to estimate the potential returns their organizations can expect from DevOps transformation:

Pyramid diagram showing KPIs that is mandatory towards Technical Metrics and Leadership Metrics
The above image depicts various KPIs and it has become mandatory for the leadership to pay attention to the indicators, especially those on the upper right-hand side.

Let’s look at some of the other KPIs that need to be considered while measuring your DevOps success.

While we have seen a list of DORA metrics, maturity assessment and the DevOps transformation pyramid, it’s also essential to know how the actual mechanism for DevOps KPI measurement is implemented:

5 Steps to Implement DevOps KPIs

Diagram showing the 5 Steps to Implement DevOps KPIs : identify, create, evaluate, change, assess

Conclusion

When all the metrics and KPIs are defined as per your business needs, along with those for long-term impacts, the developers, SREs, and the management are aligned towards one common goal with DevOps transformation being the means towards that end.  

Ozone is focused on eliminating every complexity of a DevOps team. It simplifies and automates containerized and decentralised application deployments across hybrid cloud and diverse blockchain networks. Ozone integrates seamlessly with major tools across CI, CD, analytics and automation to support your software delivery end to end for even the most complex scenarios.

Write to us at info@ozone.one