“There are many paths to success, but even more that lead to failure.” That’s one key takeaway from the 2018 version of Puppet’s annual State of DevOps Report. It’s an eye-catching statement because it spells out the reality of DevOps today: While many firms are enjoying the benefits of successful DevOps journeys, others are struggling to scale their initial efforts or to get consistent results.
Getting to the bottom of this dilemma—and helping companies fall on the right side of the divide—is a major focus of Puppet’s 2018 State of DevOps report (created this year in conjunction with Splunk)— and the same issues figure prominently in a similar report from DevOps Research and Assessment (DORA), the 2018 Accelerate State of DevOps Report. (DORA was founded by Dr. Nicole Forsgren, Jez Humble, and Gene Kim). Both reports, for example, segment firms’ DevOps efforts into performance-based cohorts. And both use these distinctions to isolate the traits that lead to successful DevOps journeys, as well as the problems that can stop a DevOps transformation in its tracks.
Critically, high-performing DevOps organizations don’t worry about tradeoffs like speed and quality. As the DORA report notes, “our research consistently finds that the highest performers are achieving excellence in both throughput and stability without making tradeoffs. In fact, throughput and stability enable one another.”
The growing gap between DevOps haves and have-nots
Of course, this raises the question: What, exactly, did the authors of these studies see when they looked at the differences between successful and unsuccessful DevOps journeys?
According to the DORA study, the distance between top DevOps performers—what they refer to as an “elite” cohort—and less-successful organizations is bigger than ever. For example, compared to low performers, the elite group has:
- 46 times more code deployments
- Lead times, from commit to deploy, that are 2,555 times faster
- A change failure rate that is 7 times lower
- A time to recover from incidents that is 2,604 times faster
The DORA study also calls out what it describes as “misguided performers”—organizations that deliberately eschew DevOps practices in favor of an extremely conservative and cautious approach to software development and delivery.
While the benefits of a deliberate go-slow approach—including more time for quality checks and testing—may sound reasonable, they’re illusory. “Developing software in increasingly complex systems is difficult and failure is inevitable,” reads the DORA report. What “misguided performers” actually get, according to the authors, is a recipe for ruin that includes change lead times of 1-6 months and taking up to 6 months to restore service following a change failure.
This growing gap between elite and low performers should be a wake-up call for organizations that have allowed their DevOps initiatives to spin their wheels, sometimes for months or even years. And pretending the gap doesn’t matter—basically stuffing your IT organization into a time machine set for 1990—is not going to work.
(Looking for more insights and best practices to help your own organization make the most of its DevOps journey? New Relic has a number of useful resources on the topic, including our eBook on Best Practices to Knock Down Barriers to Success, and our Guide to Creating an Alerting Strategy for DevOps Teams.)
Four DevOps success factors
The reports also highlight key trends that show why a successful DevOps journey is about people and culture as much as technology; and they make the case that measurement and monitoring are even more important to the success of DevOps than many people assumed.
1. The outsized role of “empowered monitoring” in DevOps success
The Puppet study uses the term “empowered monitoring” to refer to a DevOps team’s critical ability to configure its own monitoring and alerting capabilities for the services it operates.
“Empowered teams that run applications and services in production can define what a good service is; how to determine whether it’s operating properly; and how they’ll find out when it’s not,” the study says, adding, “Empowered monitoring isn’t for just ops or for some newly created DevOps team: It’s for all teams that work with technology.”
(Here at New Relic, we wholeheartedly agree. Using New Relic as a part of your DevOps initiative enables you to establish a common set of KPIs, tear down silos, and move faster confidently as a data-driven team. The New Relic Developer Program is all about fostering openness, simplification, and engagement, making it “easy for DevOps teams to get data in and out of our platform and to integrate it within their team’s existing workflows and processes,” according to Mark Weitzel,, senior director of product management for our platform and ecosystem group.)
Whatever you call it, putting DevOps teams at the center of the monitoring process is also vital to fostering accountability and ownership within an organization’s DevOps teams. It allows team members to create the kind of seamless and continuous feedback loops that foster an organic approach to accountability, rather than some external burden dropped upon them from the executive suite.
The benefits of this approach are real, and they are quantifiable. According to the Puppet study, 47% of firms with highly evolved DevOps models “are able to define their own monitoring and alerting criteria for apps and services in production,” compared to just 2% of low-performing firms.
2. Monitoring and observability are key to continuous delivery
Monitoring is also a critical part of the continuous delivery models that many view as a key metric of DevOps success.
Earlier DORA studies “found that proactively monitoring applications and infrastructure, and using this information to make business decisions, was strongly related to software delivery performance.” This year’s report, however, looked specifically at the role of monitoring and observability within an organization’s continuous delivery initiatives.
“We found that a comprehensive monitoring and observability solution positively contributes to continuous delivery,” the report reads, “and that those who have one were 1.3 times more likely to be in the elite-performing group.” The study also concluded that test automation, or continuous testing, contributed to successful continuous delivery models within many organizations.
3. The right cloud infrastructure drives DevOps success
The DORA report also found that adopting the cloud is strongly correlated with being a high-performing organization. Organizations that leveraged all five of the report’s essential cloud characteristics—on-demand self-service, broad network access, resource pooling, rapid elasticity, and measured service—were 23 times more likely to be in its elite-performer cohort.
The report also sounds a cautionary note, however, stating that “many companies in the cloud haven’t actually adopted the essential patterns that matter—and this could be holding them back.”
This essential point that has also surfaced prominently in New Relic research on the importance of getting beyond basic “lift-and-shift” cloud adoption models. Lee Atchison, New Relic’s senior director of strategic architecture, sees it as a major stumbling block for many organizations.
“Real cloud success, at scale, require much more than lift-and-shift,” Lee pointed out in a his blog post addressing The 6 Levels of Cloud Maturity. “It demands successfully navigating the world of the dynamic cloud,” including the measurement, monitoring, and cloud-unique application technologies that work hand-in-hand with DevOps methodologies and culture.
4. Monitoring and automation are key to closing the DevOps “perception gap”
The Puppet report also examines what can happen when an organization’s monitoring, alerting, and related capabilities aren’t prepared to support its DevOps journey. The issue is a gap between how an organization’s C-level executives view its DevOps performance versus how front-line developers and managers see the same issues.
“As we looked at how different roles…viewed their DevOps progress, we found that the C-suite had a much more optimistic outlook,” the report notes. “Perhaps even more depressingly, it’s quite common to hear of major differences in perception between the C-suite and folks on the ground when it comes to evaluating progress.” This ranges from the execs assuming higher usage of DevOps tools and techniques to a rosier assessment of how the organization balances its efforts to keep existing systems running against the need to work on innovations that can build an organization’s competitive advantage.
It turns out that a single, shared view of the truth about DevOps adoption and impact can make a big difference when they’re implemented with the right mix of supporting technologies. “The best countermeasures to this inaccurate communication are the mutually reinforcing pillars of automation and measurement,” the report states. “Automated systems enable better reporting of business metrics. Rather than relying on information that’s filtered upwards to executives, you have an objective measurement system to share across the business, helping everyone get onto the same page.”
The right way to nurture DevOps
Even firms who go all in on DevOps may need help nurturing a healthy DevOps culture. Fortunately, according to the DORA report, there are “a number of management and technical capabilities that influence culture, showing that you can change culture by changing the way work is done in your organization.” Initiatives that promote collaboration, sharing, trust, and autonomy, for example, benefit from technology that facilitates sharing, reuse, measurement, and accountability for results.
Like many discussions of workplace culture, that advice may seem a bit vague. The task of building a successful DevOps culture simply varies too much from one organization to the next to allow for hard-and-fast rules. But that doesn’t mean you can avoid the importance of culture in DevOps success. And there are some best practices emerging around where to begin the culture-building process: According to the DORA report, for example, one key involves promoting and supporting continuous learning within an organization.
To find out more, it’s worth clearing your calendar and digging into both of these invaluable reports. At the core of the Puppet report, for example, you’ll find a five-stage model designed to provide a universal set of prerequisites for a successful DevOps journey. That fits nicely with the DORA report’s five essentials of cloud computing mentioned above.
Also in the DORA report you’ll find confirmation that “implementing DevOps practices and capabilities during technology transformations pays off in terms of organizational performance as well as quality outcomes.” In other words, DevOps is worth the effort.