Who’s really in charge of deployments at your organization? In most companies, it’s the developers, and/or some combination of developers and QA and testing folks. But according to Cristopher Stauffer, senior director of test engineering at online marketing platform Yodle, that approach just doesn’t scale in a world of microservices.
In his instructive presentation on The Selfish Stack at our FutureStack16 Tour: New York event earlier this month, Cris explained the concept of a system specifically designed not to allow anyone, for whatever reason, to override the basic rules of what is required to be in place before deploying. As Cris described it, a selfish stack says to engineers: “I care about you, but not as much as I care about myself.”
Everything is harder at scale
Over the last 18 months, as the company moved into a microservices architecture, Cris said, it became increasingly difficult and time consuming to track and test some 1,000 deployments each month. “Lots of things that work for 5-6 applications don’t work for 500 applications,” he noted.
While Yodle had policies and best practices in place to ensure deployments didn’t cause problems, busy engineers were increasingly deploying without going through all the required steps, and Yodle’s “selfless stack” had no way to enforce the rules.
Ultimately, Yodle wanted to be able to release software updates without anyone having to approve them. The question was, Stauffer said, how can the system protect and guard against problems like incorrect configs and routing when there isn’t time to fully test all the things in an environment of constant change? “At some point things get so big that you don’t know what everything means anymore,” Cris acknowledged. “I don’t know what they are,” and that’s a good thing, he added, “because I don’t want to be a liability” or a bottleneck in the process.
Selfishness for a noble cause
“We need to become comfortable with having a landscape that’s constantly in flux,” Cris said. For the benefit of engineers and the company, Yodle needed a “selfish” system, one that would protect system integrity, he explained, and “guard our customer needs and our engineers’ time.”
Watch the video below and/or click through the slide deck to see the steps Yodle took to create a truly “selfish stack,” one that simply won’t allow a deployment that violates policy, no matter how confident the team trying to deploy it may be.
The video also shows how Yodle uses “canary” apps that test a key subset of things to build confidence that everything works, and then roll out deployments in stages—internally first, and then to just 1% of users, and then to everyone—to minimize any effects on actual users. If there are any issues, the deployment is automatically rolled back to the previous stage.
The idea, Cris said, is that “something should launch as soon as it’s ready,” he said, “not a minute later, but not a minute before,” and “we want to be able to release without anyone having to approve it.” The Selfish Stack makes that possible, he said, and represents “a huge win for engineers, who don’t have to babysit what’s going on” or designate someone to follow up on the deployment to make sure nothing went wrong.
With a selfish system, Cris said, Yodle has “gone from monolith to microservices,” with a better sense of where everything is. Because the infrastructure knows about packages pushed out to the host, if there’s a problem, the system automatically initiates instant intervention and rollback. Engineers aren’t really involved, he said, and that’s the way everyone wants it: “Don’t expect anything instant from engineers!”
Join us for FutureStack in San Francisco
For more insights like the ones Yodle shared at FutureStack16 Tour: New York, be sure to attend FutureStack16 in San Francisco, November 16 and 17. $495 early bird pricing is available until September 1: register now!
Note: Event dates, participants, and topics are subject to change without notice.