People say “speed kills,” but when it comes to software development, speed is the critical ingredient for success. If you plan for speed carefully enough, cost and quality will come along for the ride.
“Speed, if you’re serious about it, has to include both an aspect of quality and cost,” explains Mirco Hering, managing director of Accenture‘s Agile and DevOps practice in the Asia Pacific region. “If it’s not good, if you have problems and defects, you have to redo it,” Mirco says. So as long as you calculate speed by measuring when the product is actually done and correct, you’re tracking everything that’s important.
That’s the kind of insight Mirco shares in the latest episode of the New Relic Modern Software Podcast. Joining my cohost Tori Wieldt and me all the way from Melbourne, Australia, Mirco shares his opinions on dealing with legacy technology and legacy thinking and also shares the key takeaways from his new book “DevOps for the Modern Enterprise: Winning Practices to Transform Legacy IT Organizations.” (Don’t miss Mirco’s blog at notafactoryanymore.com.)
You can listen to Episode 26 right here, or download all the episodes automatically by subscribing to the New Relic Modern Software Podcast on iTunes, or wherever you get your podcasts. Read on below for a full transcript of our conversation, edited for clarity:
New Relic was the host of the attached forum presented in the embedded podcast. However, the content and views expressed are those of the participants and do not necessarily reflect the views of New Relic. By hosting the podcast, New Relic does not necessarily adopt, guarantee, approve or endorse the information, views or products referenced therein.
Fredric Paul: Mirco, can you tell us a little bit about how you came to the world of Agile and DevOps?
I started my career as a developer writing compilers for embedded microprocessors in cars. And then I joined Accenture and basically did what would now be considered DevOps in a much less sophisticated way. And then, a few years ago, DevOps became popular. All of a sudden, the obscure thing that I was doing on my projects became something that was sexy that people wanted to talk about.
I learned in some of my DevOps projects that it’s really hard to predict all the problems you have by adopting DevOps or by automating everything. A typical waterfall approach of “We’re gonna take six months to automate everything,” is not really successful. I had an Agile guy sitting with me on a project; he said, “Look, we should really take a more realistic view of project delivery. I just came from this training called Agile, should we try that?” I’ve never looked back since.
Modern software meets DevOps
Fredric: How you would describe DevOps’ relationship to Agile?
Mirco: Here in Australia, we call the practice modern engineering rather than Agile or DevOps.
Fredric: We call it modern software here at New Relic.
Mirco: At the end of the day, it’s about delivering successful, high quality products faster, right? And all organizations are a) interested in that and b) struggling with it. I really see it more from the perspective that we’re trying to help organizations improve their IT capabilities. And if some of these buzzwords help us to actually have the conversations, then all the better.
When I started this practice here locally, no one really wanted to talk to you about it. You were talking to some obscure person in the architecture team or something. Nowadays, you have CIOs and CEOs talking about Agile and DevOps, so we have to be thankful for that.
Tori Wieldt: What would you say the state of DevOps is today? Do you think we’re going to see mass adoption of it?
Mirco: I think we have mass adoption because everyone is talking about it. If you look at the amount of tools that are being sold with DevOps in the title or in the description, then it seems like everyone is doing it.
But I specialize in working in enterprises that are complex and have a lot of legacy. You have the SAP, Siebel, the mainframe—all those kind of different applications in there. And the truth is that there’s not a lot of continuous delivery in that world, and, perhaps, there never will be.
What I see now is that we don’t have to worry too much about explaining why it’s necessary, but we still have to do a lot of demystification. Is it just a matter of implementing the tool or is it just a matter of changing culture, whatever that means? And you really have to comprehensively think about what you need to do to change your organization.
Legacy technology and legacy thinking
Fredric: So, what’s your definition of DevOps?
Mirco: My definition of DevOps is everything that is required to deliver a product or a service to a customer. Terms like BizDevOps or DevSecOps or any of those additional acronyms are good to highlight the specific point. But in my definition, they’re always included.
Tori: I love the idea of legacy technology and legacy thinking. So I’m assuming that’s one of the biggest barriers you see in the enterprise. How do you counteract that legacy thinking?
Mirco: A lot of this is actually just education. Let’s take an ops example: the usual KPIs that people have used in the past for support tickets is How quickly can we resolve them? And if we resolve them, is the resolution correct the first time? That’s how tickets should be measured.
But what happens as we move into a world where we automate everything? Think about the bucket of tickets that someone gets: automation takes away the simple bits and the fast bits. What problems do you now get a ticket for?
Well, these new problems are more complex and harder to solve. That means as you increase automation and monitoring and find the easy problems, you’ll probably need a longer time to solve these bigger problems that still come through. And you’ll probably get it wrong more often than you did in the past. So we’ve created an environment where the KPIs are actually dis-incentivizing people from solving the automation part. As soon as you explain that, people are like, “Oh, yeah, you’re right.”
Tori: Oh, yeah, you’re right.
Fredric: So what are the right KPIs for an automated DevOps environment?
Mirco: One of the things that I talk about is the transition cost. That’s the cost from when the last line of code has been written for a new feature until it’s live and validated it in production. Is there any regression testing that you need to do at the end of development? Any deployment activities? Any validation activities? If you can measure that either as time—or ideally as effort—then that’s a pretty good metric for you to see how far you have to go.
But that’s very, very high level. I think there are a lot of lower-level things that you can do around specific services, like, what is the duration of your regression suite? How long does it take you to deploy the end-to-end solution? Those are smaller metrics that you can use because they’re very process-driven and they’re not too hard to find endpoints for.
Fredric: A lot of people take deployment frequency or time to deploy as a key DevOps KPI. Does that make sense to you?
Mirco: It does if you compare like for like. I wouldn’t compare Amazon with a basic enterprise because when Amazon deploys 10,000 times a day, they’re not deploying 10,000 times a day to an SAP implementation. They’re doing small components on the website. But if I make my same SAP deployment quicker or more frequent then that’s an improvement.
DevOps and the forcing function of speed
Fredric: Is that something that companies should focus on, or is it more complicated than that?
Mirco: I think that it’s useful. If you know that that’s one of your challenges, it is a useful metric. And I think speed is ultimately a good metric. There are a lot of things that you can do to reduce cost that doesn’t necessarily increase speed or increase quality.
Speed, though, if you’re serious about it, has to include both an aspect of quality and cost because if it’s not good, if you have problems and defects, you have to redo it. So as long as you calculate speed as a measurement of when a project is actually done and correct, it covers quality.
And to really be fast, you will cut out everything that is not necessary, and that means you’ll reduce cost. So it’s a good forcing function.
Fredric: In your book, “DevOps for the Modern Enterprise,” you question the whole idea of trying to cut costs and not really looking at the quality and the outcomes. Can you tell us a little more about the key themes in the book?
Mirco: There are lots of DevOps books on the market. One of the things that I asked myself was What do I add to the curriculum? So there are two aspects I try to highlight in the book that are a bit different. One is obviously that whole legacy environment that we already talked about.
The other thing is, I work for Accenture. So that means I work as a provider in many organizations. Not a lot of people have spent time figuring out what that means. If you talk about DevOps culture, it’s one thing to do that on your own team with 10 people who all have the same company badge. But big organizations have many providers. How do you create an ecosystem where DevOps can flourish? There are a couple of chapters in the book that talk about that. And I share perspectives from both sides because in different parts of my career, I’ve played a role as a client and as a provider.
Silver bullets, and DevOps in a box
Tori: Can you tell me some of the things you see that people misunderstand about DevOps? Every vendor says they can sell you a box of DevOps and we all kind of laugh about that. But what are the other misperceptions you see?
Mirco: Yeah, DevOps in a box. If only it was that easy, right? It’s a cultural thing, which is true, but you can’t attack cultural change directly. The only way to change culture is by changing behaviors and systems and everything around it. And I think we’re still looking for the right answer.
Fredric: So, is that one of the big barriers to DevOps? That everybody’s looking for the silver bullet or the easy answer?
Mirco: Yeah, and I think it’s natural. If you want to improve something, then you say, “Of course, I will do that. Just give me the business case and tell me what the outcome is.” But DevOps is not linear, right? There are a lot of different problems you need to address and it’s not necessarily predictable.
The early automation stuff is easy, and that means you can see lots of results in the beginning. And then you get to the point where it gets harder because you’ve automated 80% of your process. The last 20% is really hard. Staying the course there requires leadership beyond numbers, and that’s not easy.
DevOps in the enterprise
Tori: One common misconception is that DevOps is mainly for small startups and not for enterprises, and we know that’s not true…
Fredric: You obviously know that since your book is titled that.
Mirco: There’s a whole book about it!
Tori: So how do legacy organizations start attacking this?
Mirco: A couple of friends of mine have created startups. For them, DevOps is just natural, right? There’s no need to have a big discussion about it. It’s just how engineering is meant to be done. Great.
In big organizations, it’s a lot harder. There’s a lot more political turf that you need to cover. There are a lot more different incentives that exist in the organization, so you do to have a conscious program around it. You can make some improvements by just doing the right things, but I think you need to have a consciousness around this transformation.
I’m working with a couple of organizations at the moment where there’s a lot of good engineering but the management mindset is not necessarily behind it. So, in a big organization, I think you need to ask where you invest the engineering capability for uplift? Like, where’s your bottleneck in that sense? And how do you make sure that you’re not just tinkering? If you’re a consultant, how do you implement something that has a lasting legacy within your client’s organization. People struggle with that in my view.
Tori: We certainly see a lot of that, where large organizations have pockets of DevOps success, but to take it from the base camp to the apex of the mountain is tough.
Mirco: It’s different for every organization. I find this fascinating. That’s why I love my job—because I see all the anti-patterns out there, and they’re not always the same. Sometimes, you have really good technical engineering organizations, and every team owns their own DevOps. And then someone deletes the Jenkins server by mistake and there is no backup for it because no one actually had an enterprise-grade setup for their Jenkins stuff. “Aaaah…!”
Tori: “We were so close!”
Mirco: But that’s the fun part, right? You have to have the right balance between the super evangelistic view, the realistic enterprise view, and the multi-vendor view.
The value of DevOps in the enterprise
Fredric: What about the flip side of that? The value of DevOps for enterprises: how does that differ from its value in smaller companies?
Mirco: That’s an interesting question because it depends on how you define “value.” For small organizations, it’s necessary to survive. If you’re competing with another digital product that is coming out in the same niche, you need to be super fast to survive. So it’s crucial.
However, in big enterprises, we’re talking huge numbers. It’s not uncommon for a deployment at an enterprise to take 200 people a whole weekend—with a month of preparation. That’s millions of dollars that you can reduce. You should address the economic impact and the survival of the organization.
DevOps challenges the status quo
Fredric: You talk about the idea that you have to change behaviors to change the culture. Do you have tips about how to do that?
Mirco: I always tell people that they need to look at what is around them to understand how to change behaviors. I love the Netflix example. Not sure how true it is, but when someone is new to their organization, they deploy something into production on their first day, and if there is a problem, it shows the problem of the system, not the person. That mindset, I think, is correct.
When we find a person has behaviors that are inconsistent with the culture we want to have, let’s look at the reason why that person is making their decisions and figure out whether there’s something systematic that has forced them into that behavior. But that’s a very difficult conversation.
Fredric: To be fair, that kind of change is really difficult to make real.
Tori: Absolutely. And if you’re somebody in an organization who has spent your career building your way up in a 200-person organization that’s doing it a particular way, you have a lot invested in keeping things just the way they are.
Fredric: So Mirco, how do you take someone who’s invested in the status quo and get them to change?
Mirco: There are a couple of different ways. I often staff my team with the most skeptical person in the organization. I used to do it with the mainframe stuff. I literally took the guy into my team, and he said, “You know, we really don’t need that, Mirco. We’ve been coding COBOL for 25 years, and we’ll be okay.”
Bringing him into the team and having him work on the automated solutions provided a lot of contextual knowledge that I didn’t have, but also it was clear that if I convinced him, the rest of the developer community would be convinced as well.
Fredric: Don’t try and avoid the resistance, try and co-opt it immediately.
Mirco: Yeah, if you can. Sometimes the opposite is true as well. Sometimes, if you can’t change the people, you need to change the people, but that’s a last resort.
DevOps tools, automation, metrics, and analytics
Tori: So, a little bit about tools. Are there tools out there that can help make DevOps successful?
Mirco: Of course. I would not go back and say to just use Control-M to do deployment automation. It worked, but it wasn’t necessarily pleasant. There’s a lot of benefit in using tools, partly because they structure your automation and allow you to reuse some of it. Tools make it a lot easier to finally get metrics, right?
I’ve seen so many organizations that say they have reports for their deployments. And if you look at the deployment data, it’s hilarious because the deployments always start exactly at 6:00. That’s great because it could be triggered, but they also stopped at exactly 10:00, and you’re like, “Hmm, that’s suspicious.” It’s basically someone who manually fills in that Excel sheet—and then that’s the metrics that they use. And you’re like, “Hmm, yeah, that’s not really what we’re after.”
Having a good tool-chain that gives you good data points and allows you to identify weaknesses is great. What people need to be careful about is stuff that you’ve not automated but you still need to cover. So if you’re 80% automated, it’s not an excuse to have 80% automation in Jenkins, and 20% that isn’t automated tracked via emails.
Tori: Interesting. Talk about a little bit more about shared data across teams. Do you see that as crucial for success for DevOps?
Mirco: About three years ago, I ended up on a crusade and introduced a DevOps dashboard that we use internally with clients. That was very good because all of a sudden you could actually correlate data. You could see how your check-in frequency and your unit test coverage and your deployment frequency and the success rate of deployments related to a go-live date and the post-go-live impact on production, which wasn’t possible beforehand
If you went to the DevOps Enterprise Summit in London last month, pretty much every presentation talked about analytics.
Tori: Deciding what’s important is really key. Not measurement for measurement’s sake, but really get agreement around the table in terms of what’s a good reflection of how you’re doing.
Mirco: What you say is so powerful. I think the first instance of this kind of DevOps analytics was to just take stuff out of Jenkins and Selenium and show it. But that doesn’t necessarily have meaning. Really, we are after end-to-end data to identify where the weaknesses are.
Tori: That’s how you get teams working together.
Fredric: How do you convince enterprise leaders that they need to do this?
Mirco: To be honest, there’s not a lot of convincing them to do DevOps. But there are two conversations that I see these days. One is, Do we actually show them that they are not there yet? There’s quite a few people who believe that because they have a Jenkins server, they have DevOps, which sometimes is an uncomfortable conversation to have.
The other one is, Is DevOps feasible here? And I think that’s where a lot of this legacy thinking comes in.
The industry has not done a great job around this. You talk to a big bank about DevOps and they say, “Well, we don’t want to deploy 2,000 times a day.” Yeah, but that’s not the point. The point is you do it better than what you’re doing right now. It’s really helping them identify the pain points. From there on, the conversation becomes easier.
Fredric: That makes a lot of sense. Any final thoughts you’d like to share with our podcast audience?
Mirco: Here’s my parting thought for success in DevOps:
I was sitting down with a whole bunch of DevOps gurus and asked, “What do you look for in organizations when you engage with them for the first time to see whether they will be successful?”
In the end, we all agreed on one thing: If you can find an organization that understands how to judge progress and has the leadership to actually stay rigorously on path, they will be successful. So if you’re asking me how to be successful with DevOps, it’s understand what your problem is, understand how you can measure progress, and then stick with it even when it gets hard.
Note: The intro music for the Modern Software Podcast is courtesy of Audionautix.