Scale vs. efficiency.
According to Sandeepan Banerjee, ClusterHQ’s new senior vice president of engineering and operations, that’s one of today’s biggest conflicts for modern dev and ops leaders. And he comes down clearly on the side of scaling.
In the latest episode of The New Stack @ Scale podcast, The New Stack’s Alex Williams and I talk with Sandeepan—who joined ClusterHQ in January after spending a decade at Google working on Web search infrastructure, YouTube’s video infrastructure, and as “Data Head” of storage infrastructure.
As Sandeepan puts it, “given the relentless logic of Moore’s Law,” the biggest opportunity in the new stack is scale, often replacing the “traditional, historical focus of IT” (time code 9:22).
“I don’t think we are saying, ‘Go on and be inefficient,’” he continued. “It is more that the opportunities that you can get from users and their satisfaction from exploiting scale are more numerous in the environment today than the opportunities you can bring to your organization from being more efficient or parsimonious in your execution of source code.”
Put simply, there’s a lot more to gain from quickly taking advantage of the power of increasingly cheap and flexible computing cycles than there is trying to save a few bucks by maximizing the value of every cycle you use.
Sandeepan’s example? “The Y2K problem, where somebody tried to save two bytes” by using two digits instead of four digits to represent the years. “Yes, it is the efficiency decision,” he said, “but it’s not particularly bright in the historical context.”
Culture and technology = fighter pilots and jets
Scaling vs. efficiency isn’t the only dichotomy Sandeepan explored. He also talked about the parallels between the culture of scaling and the technology involved. Both DevOps and the concept of breaking engineering groups into small teams mirror the movement to replace giant monolithic apps with fleets of microservices. Both present the challenge of managing and integrating multiple parts into a seamless, coherent whole (21:38).
And make no mistake, the culture is just as important as the technology. “Just as if you are a fighter pilot, you are sort of married to your fighter jet—you have to understand this as a continuous human-computer interaction. In the same way, the culture and the technology in the modern data center are not really separable. You cannot take [just] any technology and make [just] any team operate it, and vice versa” (26:05).
All this and much, much more in the latest episode of The New Stack @ Scale Podcast:
New Relic is a sponsor of the New Stack @ Scale Podcast. However, the content and views expressed are those of the participants of the New Stack @ Scale Podcast, which is the property of The New Stack. Any views expressed on the New Stack @ Scale Podcast do not necessarily reflect the views of New Relic. By embedding the audio for the New Stack @ Scale Podcast or linking to The New Stack, New Relic does not adopt, guarantee, approve or endorse the information, views or products available on The New Stack site.
Read more of Sandeepan’s scaling insights in Joab Jackson’s post The New Stack @ Scale Podcast: ClusterHQ and the Science of Scaling