Waterfall development is so over. These days, most software development organizations realize that some version of agile development practices are pretty much the only way to create high-quality software quickly enough to keep up with the ever-increasing demands of consumers and businesses. That’s not news anymore.
But the growth of agile raises some critical questions, including, What does it really mean to be agile at scale? How do you manage data and understand your own software stacks and how they correlate to your IT operations? Just as important, how does that affect individuals and the overall architecture of the organization? That’s the worthy subject of the latest episode of The New Stack @ Scale podcast, the New Relic-sponsored show about developing and managing elastic and dynamic services and systems.
For this latest episode, host Alex Williams, founder and editor in chief of The New Stack, and I welcome Daniel X Moore and Eric Sammer. Daniel works at Fog Creek Software and is a developer on HyperDev, a new tool that lets people more quickly create and deploy applications. Eric is CTO at Rocana, which builds “large scale IT visibility in operational systems, mostly around super high volume log metric and other kinds of business transactions or event data within the data center.”
So Daniel brings a frontend perspective, while Eric focuses on the backend, with the Rest API layer as the logical dividing point. With those two viewpoints, our discussion delves swiftly and deeply into the meaning and implications of agile at scale. Here are the 10 most interesting points we addressed:
Agility has many meanings, but iterative change is key
“Agility could mean all kinds of things,” Daniel says. “It can mean you are a certified scrum master and you follow the exact process dictated by agile … but in my opinion, it’s that you make a change. You get something out in the world. You get some feedback from the world and then you can make another change.”
Agile doesn’t mean breaking things
“Oh, please don’t put it in production where it’s going to wake me up at night,” Eric responds. To him, Agile means “continuous integration and continuous deployment”—democratizing access to that process and making it repeatable and consumable without spawning all kinds of crazy errors. “Facebook’s ‘move fast and break things’” is great, he says, “Unless it’s the payment system, unless it’s privacy controls, unless it’s information security, and all these other kinds of things.”
Applications and systems scale, not individual slices of technology
Developers get into a whole lot of trouble when they start thinking about parts of the system rather than the complete and sort of holistic view of the system. As Eric puts it, “When you say, ‘technology XYZ scales and technology ABC does not,’ the reality is that … applications and systems scale, not individual slices of it. So if you used the most scalable thing in the world on the frontend and a single monolithic system or application on the backend, you’re sort of in hot water, right?”
Scaling agile can mean bringing development to new users
According to Daniel, his HyperDev app-creation tool is an example of scaling agile development to new parts of an organization. “What we found in developing HyperDev is, as the barrier gets lower and lower, more people in the organization, whom you might not traditionally think of as developers, are able to contribute, are able to build applications, are able to solve their own problems, [even if] they might not consider themselves developers.”
Agility in the enterprise
“My experience is very much sort of the enterprise world—data management and data warehousing and stuff,” Eric says. “I think about Fortune 100 customers tripping over themselves to build systems or customer experiences like Facebook and Google and Twitter. They’ll all tell you, ‘We want to work like that,’ but their internal infrastructure, their development methodology, their time to market, everything about how they run their business [doesn’t support that].” In that environment, agility is not just about how to get an individual productive … but about how do you get a team of people to be able to take advantage of what another team within the organization is doing?
The value of a single platform
“From my perspective,” Eric says, “it’s all about platforms that allow people to build applications that are operational, can be easily operationalized to deploy to production, can be easily scaled up.” In fact, he adds, “You could argue that a lot of the internal standardization and sort of technology-first attitude of a Google or a Facebook … actually enables agility in a way that I think corporations that are large and sprawling don’t quite have access to.”
Siloed organizations develop siloed products
“The software you develop, the tools you develop, are a reflection of your organization and a reflection of your process,” Daniel says. “If you have a very siloed organization, you’ll develop very siloed products and have very clean distinctions. If you have a homogenous organization, there might be fewer boundaries between your tools,” which could make for more agility in terms of deploying things quickly.
Agility gets harder at extreme scale
“Agility is super easy for five engineers sitting around the table,” Eric says. “Agility is super painful when it’s a thousand engineers across geographic boundaries and stuff like that with different skill sets.… Having standards, making it easy to do the right thing, is super critical.” For example, he adds, “New Relic makes it easy to instrument your applications. So why wouldn’t you do it?”
Agile is about more than just tools
“As developers,” Daniel adds, “we have almost an unlimited number of tools and they are getting easier and easier to integrate.… But if you don’t have the culture and the process and the experience of ‘How do we use these things?’ … Technologically, we have the capability to have all of these tools. They exist. They’re out there. You can find them. You can hook them up. But can you understand them? Can we learn from them?
Agile is a human problem
Companies like to say, “Well, I’ll buy this technology and then I’m agile.” Nope. “This is just as much a cultural shift and organizational shift as it is a technology shift,” Eric notes. The question is “How do you maintain standardization and cohesiveness in an environment where you’re trying to give teams and individuals’ autonomy around technology?” Eric asks. “It goes into sort of psychology, sociology, how do you structure your company,” Daniel adds. ”If your structure is, one guy is in charge and all the other people do what he says, then all of your systems, all of your process, all of your tools are going to mirror that structure. If it’s that anyone who has a good idea can get something out there, then your tools and your process are going to mirror that. … You can’t just inject technology and change things. You have to create the system, create the structure within your organization if that’s the result you want to achieve.”
There’s much more in the podcast, of course, including the invention of “Agile in a Box” and an automatic download of DevOps. Be sure to listen to the whole show to get Eric and Daniel’s key takeaways for being agile at scale.
New Relic is a sponsor of the New Stack @ Scale Podcast. However, the content and views expressed are those of the participants of the New Stack @ Scale Podcast, which is the property of The New Stack. Any views expressed on the New Stack @ Scale Podcast do not necessarily reflect the views of New Relic. By embedding the audio for the New Stack @ Scale Podcast or linking to The New Stack, New Relic does not adopt, guarantee, approve or endorse the information, views or products available on The New Stack site.
For more information and another take on the podcast, see Kiran “CK” Oliver’s insightful post: The New Stack @ Scale Podcast, Show 11: Agility Inside and Out.