Guest post originally published on Ozone’s blog
Preface
Everyone talks of DevOps and how their organizations are going about transforming their development and operational processes. Investments are high and measuring the real-world outcomes of these DevOps transformations is quintessential. Towards this, we have the DORA metrics. However, in this article, we highlight why this is inadequate and where the gap lies.
While DORA metrics focus on technical KPIs, how does an organization measure the long term business impact? For example, if you automate your testing and shift-left security scans or invest in securing your releases, there is no doubt time and effort saved while improving on quality. But has this really impacted on the long term?
The need to measure DevOps performance
You’ve heard the term “DevOps transformation” a number of times but ever wondered how it plays a role in your business achievements and why should you leverage it?
A DevOps transformation requires a significant amount of time, money and capital to an organization, necessitating it to gauge whether its resources and efforts are taking shape in the right direction to give the expected business results.
But realistically, every team requires different resources for different needs. Value stream mapping here helps to a great extent. In that, it helps increase visibility throughout the CI/CD stages, hence making it easier to visualize and map value to be delivered across from code to customers. A Value stream delivery platform (VSDP) saves time, skills and reduces the costs by half! One such VSDP is OZONE that not only has these features but strongly focuses on enterprise value, providing higher quality to their customers with impeccable performance.
Coming back to the topic at hand, let us see what exactly is DevOps transformation and the areas where DORA metrics fall short of.
DevOps Transformation
Transforming DevOps means bringing in lean thinking and automation together with latest tech, practices, and tools leading to better business outcomes.
The main DevOps goal is to optimize the flow of the value from idea to end-user. Most software and IT Operations teams have adopted DevOps practices to evolve faster. Still, there are companies that are under pressure to adopt it in order to bid/compete with the rival companies, innovate faster and deliver better digital experiences.
Typical challenges faced by DevOps adopters:
- Multiple teams have different tool preferences based on various clouds, frameworks, and use cases. Implementing aspects like testing (stress, load, security scans) in such a scenario across DevOps stages would add to the challenge until the product hits MVP, which in turn effects stability
- Resistance to change the mentality of Developers vs. Operations towards a shift-left thought process to quicken the delivery pace
- Establishing a common standard of CI and CD practices
- Siloed team culture
- Gaining Multi-cloud skillset
- Collecting metrics for effective measurements
The list of challenges is a mix of various issues, but they can all be overcome by focusing on people, processes, and technology. The boundaries between the business, development, and operations have to be removed by developing a robust DevOps transformation roadmap. However, when it comes to adopting the DevOps philosophy, the most critical factor to all the organizations remains the same, i.e., quantifying the long-term benefits of DevOps. Yet, this still seems to be overlooked and is lost in the process.
While teams focus on DORA metrics, it is still not sufficient to capture these long-term benefits of DevOps. Let us first look at what these metrics are followed by how the shortfalls can be tackled.
DORA (DevOps Research and Assessment) metrics help make performance measurement easy and make it available for teams to consume it and take actions wherever needed.
DORA’s four eyes that help you measure DevOps performance:
- Deployment Frequency – How often an organization is pushing code to production
- Lead time for changes – How long does it take from code commit to code being deployed to production
- Mean Time to restore service – The amount of time an organization needs to recover from the failure
- Change failure rate – How many percentages of deployment are causing a production failure causing the user-facing issue
DORA has set the DevOps standards using which the organizations can identify the metrics for Elite, High, Medium, and Low performing teams. According to this, Elite teams are twice as likely to meet or exceed their organizational performance goals. A detailed visual below from the 2021 State of DevOps report:
Why is DORA not sufficient to show the big picture?
DORA metrics have become a key indicator to check the progress: It is easy to fetch data for these metrics and it is often believed to be the sole indicator. But let’s not forget that high-performing organizations also look at customer insights and delivered values even more so than these metrics. .
Moreover, relying on one set of metrics for various products and teams is challenging as every organization is different. As there are structural differences between you and other organizations when it comes to products, team size, adaptability and experience of the team.
DevOps is a change of mindset rather than of technology. Hence, according to Gartner’s report, through 2022, 75% of DevOps initiatives will fail to meet expectations due to organizational learning and change issues.
Because of this, dealing with mere metrics is not done. Organizational learning and team dynamics may not show their impact on DORA metrics but may very well lead to better business outcomes, which contradicts DevOps measurement!
Hence, DORA metric alone cannot show you the big picture or the long term benefits of DevOps transformation.
Let us try moving ahead of DORA and look at the DevOps maturity assessment of an organization and see if this helps show the long-term impacts.
How is DevOps Maturity assessed?
DevOps Maturity is considered a model that dictates an organization’s position in the DevOps journey. It defines the current status and what needs to be accomplished to achieve the desired results. DevOps maturity is measured by keeping the following pointers in mind;
- Culture and Strategy
- Automation
- Structure and process
- Collaboration and sharing
A perfect DevOps maturity model operates on three levels:
- Assessing the current state of capabilities
- Identifying areas of improvement
- Outlining steps to achieve desired DevOps goals
To be able to determine the maturity levels, the DevOps must comprise three components:
- DevOps Maturity for Application
- DevOps Maturity by Data
- DevOps Maturity by Infrastructure
5 Stages in DevOps Maturity Journey
While the stages give you a complete picture of DevOps status in the organization, this maturity model comes with other benefits too!
- Prompt adaptability to change
- Ability to find opportunities
- Improved scalability
- Operational efficiency
- Increased delivery speeds
- Enhanced quality
With the DevOps maturity model you can improve organizational workflow, enhance release frequency, and make production faster.
Other metrics that help to quantify long term ROI of DevOps transformation
Checkout some of the crucial areas of analysis that leaders can use to estimate the potential returns their organizations can expect from DevOps transformation:
- Time taken with automated deployments vs manual deployments with the older approach. This highlights the ROI for the efforts saved in turn saving the engineering efforts.
- Evaluate the cost saved by reinvesting that saved time and energy in new features and meeting unique customer needs by focusing on value delivery
- Measure how DevOps help in cutting down costs and risks by reducing time in resolving issues, avoiding downtime, and maintaining security and quality
Let’s look at some of the other KPIs that need to be considered while measuring your DevOps success.
- Availability and Uptime
- Time and Frequency of Deployment
- Deployment failure rate
- Meantime to failure
- Mean time to detection and recovery
- Change failure rate
- Lead Time
- Early detection of defect
- Customer feedback
- automated test case
- Application performance
- Process cycle time
While we have seen a list of DORA metrics, maturity assessment and the DevOps transformation pyramid, it’s also essential to know how the actual mechanism for DevOps KPI measurement is implemented:
5 Steps to Implement DevOps KPIs
Conclusion
When all the metrics and KPIs are defined as per your business needs, along with those for long-term impacts, the developers, SREs, and the management are aligned towards one common goal with DevOps transformation being the means towards that end.
Ozone is focused on eliminating every complexity of a DevOps team. It simplifies and automates containerized and decentralised application deployments across hybrid cloud and diverse blockchain networks. Ozone integrates seamlessly with major tools across CI, CD, analytics and automation to support your software delivery end to end for even the most complex scenarios.
Write to us at info@ozone.one