Conway’s Law. The relationship between agile and DevOps. Autonomy vs. alignment. Inversion of control. The false promise of “loose agile.” The difficulty of assessing team health. The critical role of change management. Why you don’t want to buy a “box of agile.” The surprising influence of physical architecture on team productivity. The importance of flowers.
These are just some of the important topics that New Relic Developer Evangelist Tori Wieldt and I discuss with Brent Miller, a principal engineer and architect in New Relic’s Portland engineering headquarters, in the latest episode of the New Relic Modern Software Podcast.
You can listen to the episode below, get all the episodes automatically by subscribing to the New Relic Modern Software Podcast, or read on below for a full transcript of our conversation, edited for clarity:
New Relic was the host of the attached forum presented in the embedded podcast. However, the content and views expressed are those of the participants and do not necessarily reflect the views of New Relic. By hosting the podcast, New Relic does not necessarily adopt, guarantee, approve or endorse the information, views or products referenced therein.
Fredric Paul: Brent, can you give us a little sense of what your role is at New Relic and what your background is, how you came to us?
Brent Miller: My background is a bit unusual. I don’t come from a comp sci background. My training is actually as a plant evolutionary ecologist. I’ve got a master’s degree in studying the evolution of plant sex. (Flowers are awesome and they’re super important because they make most of our food.)
My role at New Relic right now is I’m organizational architect. When I joined the company, I was a UI engineer and over the years, as I got better at engineering, I eventually became an architect and then as we went through our big reorg two years ago called Project Upscale, I learned that I have a real passion for this kind of organizational design work. And so about nine months ago I pivoted from being a technical architect to being an organizational architect.
Fredric: What exactly is an organizational architect?
Brent: That’s a good question that I’m still trying to answer. If you’re not familiar with it, Conway’s Law states that the software you produce will be a reflection of the communication patterns inside your organization that’s building the software. The famous example is if you have four teams building a compiler, you end up with a four-pass compiler. As the architecture team at New Relic, we own the assignment of technical responsibilities into teams. Because of Conway’s Law, if we expect good architecture to be produced, we have to have the right ownership to produce the right kind of architecture. And so the work that I do is at the intersection between the humans, the organization, and the code. It’s playing in all three so we can optimize the outcomes.
Fredric: I think it’s fascinating that we’re facing that head-on. Do most organizations do that?
Brent: My experience is no. I think there’s a lot of folks in the industry who are thinking about these problems, but very few companies that are actually tackling this problem head-on and really trying to do something good with it.
Fredric: With that in mind, let’s get to the subject of today’s show: being agile at scale. What does that phrase mean to you?
Brent: Agile is actually a very interesting topic. There’s a lot of debate about what it means. People talk about Scrum, they talk about Extreme Programming, they talk about Kanban and those are all different aspects of it. To me, agile is about very tight feedback loops and a strong sense of self-determination. If you have those two things, it enables you to make many decisions in a very rapid time frame.
If you contrast it with the way that we used to ship software as an industry—which is very much waterfall—a project would take two years to finish. Well, by the time the two years are up, you’re shipping something that was designed two years ago. Is it still the right solution for the market that you’re in today?
Tori Wieldt: Was it ever?
Brent: A great question. In an agile mindset, what you do is instead of saying, “Let’s ship what we can ship in two years,” let’s say, “Two weeks. What can we ship in two weeks?”
And so every two weeks you get an opportunity to check in, see what you shipped, see if it’s good, see if you’re in the right direction, and modify your plan as you go. By making many, many small decisions, you actually mitigate the risk of the big project and you’re able to deliver something every two weeks that works.
That is what agile is about and at a team level, it’s very easy to see how that would work. At an organizational level, it’s actually the same. It’s about that learning mindset, very short iterations, and adjusting as you go.
Tori: The other buzzword du jour is “DevOps.” Tell me how agile and DevOps relate? Can you do one without the other?
Brent: To my mind, at least, they’re actually very much orthogonal. They’re very different things and I think you get the best results if you do both.
DevOps means breaking down the traditional wall between a software development team and an operations team—you have the people who build the software operate the software. That means that if you’re writing the business logic that’s handling the high throughput transaction, you’re going to make it efficient because you don’t want to get paged in the middle of the night. In the pre-DevOps world, somebody else would get paged, so you didn’t care. DevOps is super helpful. It produces much higher-quality software.
Agile is about tight iterations and tight feedback loops and you can do that with or without DevOps. I think they’re very complementary to each other but you can do one without the other.
Fredric: Can you talk a little bit about our experience moving toward a more agile architecture?
Brent: Before we had our agile transformation, we were doing what I call “loose agile.” Our teams followed a lot of the agile ceremonies: we had daily stand-ups, we would do demos, that kind of stuff. But we didn’t necessarily have the learning culture in place to support the true agile flow.
As we’ve gone through the transformation, we’ve had to double down on closing those feedback loops and really generating the learning culture that we need for the agile process to be effective. There have been lots of questions about who shows up to the demo, how do they give feedback? Let’s really dial in on that and we all have to leave our egos at the door so we can take the feedback. There’s a lot of both personal and organizational growth involved in making the transformation happen.
Fredric: You talked about Project Upscale, which was our big reorganization a couple years ago where we realigned all of our teams and let our engineers, actually, required our engineers to choose the teams that they were going to be part of. Can you talk a little bit about that and how the results of that has helped us chase the twin goals of being both autonomous and aligned at the same time, which seems oxymoronic to me?
Brent: There was a lot of thought that went into the organizational design that we put into Project Upscale, and there were a few goals we were trying to accomplish.
First was an inversion of control. In a typical organization, executives make decisions and they communicate the decisions down the hierarchy to the local teams. The local teams use those decisions, then build code.
In the inversion of control system, the local team is making the decisions. This is a much better solution because they are the experts on the problem space. They’re closest to the problem. The executive who’s up at the 100,000-foot level doesn’t have the same understanding of the problems. With an inversion of control, the people on the ground make the decisions, and they communicate the decisions up to the execs.
Fredric: That’s the autonomous part. What about the alignment part?
Brent: In our pre-transformation system, decisions moved down and information moved up. You have to flip that around. So the information has to move down from the executives. So the exec’s job now is to provide the context that the teams need to make good decisions.
They provide information on, for example, what is the competitive landscape? How is our go-to-market team taking our software into the field? They provide that information to the local teams who can then make great decisions because they have all the information that they require. The autonomy is the local decision-making, but you have to have that alignment around the North Star. The executive’s job is to provide a good goal for teams to all pull towards—that’s how we balance those two aspects.
Fredric: Are there any lessons that would be useful for other companies going through what we went through, even if they’re not planning on a Project Upscale-style realignment?
Brent: There’s a few things we learned here. First is you have to have everybody bought into the change before the change comes. They have to be on board that change is coming before you even start discussing what the change is going to be. Change management is the heart of any agile transformation, and if you don’t get your change management right, your transformation will ultimately fail.
Another lesson we learned is that when you start talking about terms like “team autonomy,” that is a very difficult term because everybody’s going to have a different vision of what that term means.
Fredric: I think it means letting me do whatever the heck I want, which could sow problems throughout the organization.
Brent: That’s what I like to call independence, not autonomy. The analogy I use, being a biologist, is to look at all the cells in your body. I’m sitting here having a conversation with you and I’m moving my hands around. And all 10 trillion of my cells are working in concert to make this possible.
That’s only possible if they are autonomous but are respecting information flows that are coming in from their external environment. Your nervous system is signaling to the musculature. The muscle cells are taking those signals and doing something with it. They’re also doing their own stuff. Each of those muscle cells has its own metabolism. When we have cells in our bodies that cease to act autonomously and start behaving independently, that’s called cancer. That’s literally what cancer is—cells that are no longer respecting the signals from the outside and so start reproducing without bound, and you get a tumor.
Independence and autonomy are very different. We’re doing a lot of work right now to formalize where the boundaries are for the autonomy; what things teams have control over versus what things they just have to do because we’re part of a larger system.
Fredric: Within that, how do you encourage the teams to create positive change, to be autonomous but not necessarily independent?
Brent: Some of that is process. You know, you create processes that clarify what decisions people can make. But what we’ve really found helpful is our new internal Product Organization Handbook.
Tori: I got a peek at it. I’m impressed. It’s cool.
Brent: It’s big, but it describes our best thinking about how the organization ought to run. We’re trying to make the implicit rules about where the autonomy boundaries lie explicit.
Fredric: Is this is something we’re going to publish for the world at large at some point?
Brent: Possibly. If you recall the “Valve Handbook” became a big thing. That’s a thing we could do with our Product Org Handbook. It represents our best thinking about how good software product gets shipped.
Fredric: That actually feeds into my next question. How do we tell how well we’re doing at this?
Brent: A lot of it is reflected on the execution side. Before our agile transformation, we were working up to our big annual user conference, FutureStack, and we were having a really hard time getting a compelling demo together.
A lot of our teams were gridlocked. They had mutual dependencies. We had diamond dependency problems that landed on our site engineering teams in particularly a hard way. Nobody could get anything done. And then we did our agile transformation, which took a lot of time. When it was done, though, we not only paid ourselves back for that downtime but got ahead of where we thought we were gonna be for the next year’s FutureStack.
You see it on the execution side. Velocity picks up, you start shipping more stuff. That’s really the primary metric: As a product org, we deliver value by delivering software, so look at the software delivery to see how you’re doing.
Fredric: But that’s only one of the issues. The other issue you mentioned is team health. How does that play into it?
Brent: It’s our firm belief that a team produces better work than a single person. So we now task teams with work instead of individuals.
But being a team’s hard, right? At an early-phase startup, as an engineer, I would get a project and I would do the project from start to finish. It was fantastic. No coordination needed. There’s very little communication needed because it’s all in my head. I can just do the thing, right?
But if you’re in a group of 4 or 5, 6, 7, 8, 9, 10 people, then it’s a very different game. You have to build trust with each other. You have to act as a unit. Building a solid team is very important for the success of our system because the teams are the bedrock of our system.
We were looking at our system and realizing that not all of our teams were doing so great. So we decided to figure out what was going on and produce a systematic way to assess team health so that we could understand where to focus our efforts.
Prior to that systematic assessment, some teams would be perceived in trouble or actually in trouble and raise their hands to say, “I need help”—and execs would dive in to help. They mean well. The net results are not always what you would want, though, in that scenario. So we’re trying to set up guidelines and build team health around a set of standards.
Fredric: What are those criteria? Is it just productivity? Are there softer measurements?
Brent: We’ve broken down those indicators into six major themes: healthy management, healthy software, healthy culture, focus on value, effective delivery, and self-determination.
The first three are very much about what makes a healthy software team and those are applicable pretty much anywhere. It’s things like, we have a culture of inclusivity, everybody can participate in meetings, that sort of thing. Healthy software is, do you have too many pages, is your stuff reliable, do you have past-due security vulnerabilities? That kind of stuff.
The second three—effective delivery, focus on value, and self-determination— are about our very particular organizational design. Do you have teams that are focused on business value when they’re making their product management decisions? That’s what that first category is.
The second one, effective delivery, is do you have good software development practices in place such that you can ship high-quality software on a regular and frequent cadence? The third category, self-determination, is do you have the knowledge that you need on your team to make good business decisions?
We achieved that goal by putting product managers on the teams during our reorg, but just having a product manager on the team doesn’t mean that you’re actually making your own decisions. We’re really trying to assess whether it’s working the way we expect it to.
Fredric: So since we’re New Relic, we have to ask, what’s the role of data and monitoring in this process? How do we track how well we’re doing?
Brent: With my science background, I love all the datas. We generated data—some of it quantitative, some of it qualitative—from all of our teams, and in an anonymized manner aggregated that data up and we’re able to look at a picture of our full system. It’s really fantastic because when I presented this data to our organizational leadership, they looked and they said, “Oh, I understand where the problems are. Here are the top three things that we want to work on.” We’re able to drive positive change into the system that way. I say “positive change,” but we don’t know yet if the change is good because we’re just now executing. But we’re able to understand the shape of our organization overall.
The second way we’re able to drive positive change is with an assessment, which is a very detailed discussion with the team to facilitate a self-assessment for every team. After that discussion we asked every team to produce a health plan for themselves. So we asked them, “Where are your strengths? Where are your weaknesses? Where do you need help? What are you going to do to make your team healthier over the next six months? What are the things that you can do that are fully within your own control?”
After asking teams to do this, every team is driving change for themselves. And at an organizational level, we’re driving change into the system. It’s a lot of change actually. That’s why we’re only doing this once every six months because if we did it more often than that, it would be too much.
Fredric: So it’s not a constant thing, it’s a snapshot?
Brent: For now, yes. I have in my mind a long-term design where we can check the pulse of teams every week or two in a very lightweight way. One thing you want to do with a team health assessment like this is to understand before a team melts down that they’re heading for a meltdown. If you’re looking only once every six months, you’re going to miss most of them. Congrats, we missed it, right? (Plug for New Relic Agents, high-velocity sampling is the thing!)
Tori: On my last visit up to Portland, I tagged along while you gave an office tour to a customer, and you talked about how you’re changing the physical alignment of the office to help teams. Will you talk a little bit about that?
Brent: Pre-Project Upscale, the focus was on the individual. Individuals were tasked with work and we typically had this open seating arrangement that you see everywhere. It’s a terrible seating arrangement. It leads to absenteeism because diseases spread. It’s hard to focus because of acoustic interference.
As we focused on teams and started building out new space, we thought, “Let’s actually build team spaces. Let’s build our space out to help our teams to be more successful.” So we started building team rooms into our spaces in Portland—there’s a room with a door that shuts. This is important because it keeps the noise away. It gives the team a space with walls that are permanent so they can put things up on the walls. If your team has working agreements, you can put them on the wall. If you’ve got a big goal that you’re driving for, you can put it on the wall.
You can create an information-rich environment. All of the rooms also have two TV monitors that can be hooked up to your monitoring software so you can see your New Relic dashboards. You have whiteboards in the room. You don’t ever really need to leave your room for anything. If you have a team meeting, everybody just turns around, faces the center of the room and they’re having the team meeting.
We’ve also coupled the team rooms with a little huddle room. The huddle room has a full AV setup for video conferencing and will hold about half the team. If you’re having a one-on-one with your manager, you just go to your huddle room. You need to go pair on something with one of the engineers, you don’t want to bother the other people in your space, go in there. If your team does mob programming, you just use the main space. We’re trying to create an environment where those teams can be very self-contained and have an environment that really supports the best work that they can do, which is about fostering collaboration and communication.
Tori: Wow, that’s cool.
Fredric: Are there any other conclusions that we’ve reached that would make sense to share to our listeners?
Brent: Agile is hard. Your agile transformation is never done. It’s like security; you don’t ever check a box on your software and say, “My software is now secure.”
It’s an ongoing process and as you adopt the agile mindset, it’s about the continuous learning and the fast feedback loops. Your system is going to change more, not less. People have to be ready for that.
Fredric: So you’re telling me I can’t just order a box of agile?
Brent: You can order a box of agile! There are many vendors that will do that for you. But a box of agile will never really fit very well for you because it will not take into account your particular circumstances, your culture, your people. A really good agile consultant—and there are lots of them out there—will come and work with you to develop the box of agile that’s right for you. So you can have your customized box.
Note: The intro music for the Modern Software Podcast is courtesy of Audionautix.