Over the course of my career as a software engineer and crusty old UNIX curmudgeon, I’ve been involved in a lot of technology decisions. Some of them have gone well, others not so much, but over the years a distinct pattern has emerged: boring technology is awesome. It may be less shiny, but boring technology lets you spend all your time writing code and building new things that actually work. And that’s the most exciting part anyway!
On the other hand, sometimes you really do need something new. With that in mind, we’ll also explore situations where the benefits may outweigh the risks of using shiny new technologies, and discuss strategies to mitigate those risks when it does turn out to be warranted.
Boring technology is best technology
Back when I was a sysadmin, we had a saying: “Standard is better than better.” Standard technologies are stable and well understood. You can usually find the solutions to problems on Google or Stack Overflow. It’s also easy to find engineers familiar with standard technologies–in fact, your current engineers probably already are. Most importantly, the standard technology will be the default assumption of every other component in your stack. Go with the standard, and you can usually just drop it in and watch it work.
The new hotness can burn you
On the other hand, it’s easy to assume that an exciting new technology is going to solve all of your problems. Most of the time, though, that new hotness only looks so alluring because you don’t know the downsides (yet).
Beside the myriad random one-off difficulties that plague most untried technologies, there are a few universal problems with new technologies that you can count on:
- None of your coworkers will know how they work.
They likely won’t have experience with basic APIs and usage, let alone under-the-hood knowledge.
- APIs can change out from under you.
We’ve run into that here at New Relic with Docker–partway through building our custom infrastructure, the APIs changed, and we had to do a fair bit of reengineering.
- There’ll be fewer established standards.
Coming up with your own standards for things like naming conventions, directory structures, and so on will often involve a lot of bike-shedding.
- They’re less secure.
They’ve probably had far fewer security-savvy eyeballs on the code than a more established technology.
- They lack basic tooling.
Even if there is tooling, it will be newer and even less well tested than the technology itself.
- They’re going to be buggier, and they’re going to be weirder.
None of the rough edges will be smoothed out, yet.
- They might turn out to be Mongo.
Now, I don’t want to be unfair–Mongo DB is a fine solution for a very specific set of situations. Those situations were not well understood when it first came out, though, and a lot of engineers got swept up by the excitement of a new technology and jumped on the Mongo bandwagon. Down the road, many of them found that their data actually was fundamentally relational. They needed to do queries across different types of data, and that data didn’t nest well in a single document hierarchy. This killed a lot of projects, and even those that successfully switched to a standard relational database paid an enormous price in doing so.
When to use shiny tech
That said, sometimes you do need to use something new. The most important factor in making that decision is understanding the context. There’s an entire spectrum of acceptable risk, depending on your situation.
On one end, you have personal side projects. Go nuts! Use every shiny new thing you can find! You’ll have a lot of fun, and you’ll encounter a lot of useful new ideas and approaches. Make sure the new technology solves the problems you think it does, and consider how you might test it. Remember that you’re not just going to be an advocate for this technology, but also a teacher, so prepare yourself for that.
In the middle of the spectrum are things like internal tooling. These represent a great opportunity to try something new and see how it works in your environment. An employee directory would be a good example—it’s a fantastically useful thing to have, but it won’t cost your company any deals if it goes down for a few hours. A third party product like New Relic also falls into this category, since you can easily point it at a few internal applications before deciding whether to roll it out company-wide.
Customer-facing production code
At the opposite end, you have customer-facing production code. If somebody is paying you to build a stable system that will be maintainable for years to come, you should have a very good reason to deviate from standard technology choices. There has to be an actual, specific problem to be solved that trumps all the extra investment of a new technology, and you should have a clear sense of what that investment will cost.
“It’s a better fit” is never a good enough excuse. If somebody says this to you, challenge them on it. If you are tempted to say this, think a little harder about exactly what problem the new thing solves. If you do, you’ll have a much stronger argument for the new technology. On the other hand, if you can’t come up with a compelling reason, you should probably stick with the standard.
When it’s worth it
So, when should you use an exciting new technology in production? Sometimes, there simply isn’t a boring solution to your problem. I once worked at a company where we were analyzing some very large quantities of network data, and our PostgreSQL-based solution just couldn’t keep up. We ended up going with Hadoop, and we paid the price for it. We spent many, many man-months learning Hadoop, building our cluster, and putting together all the infrastructure to actually use it. At the end of the day, though, it let us do something we simply wouldn’t have been able to do otherwise. For us, it was worth it.
Other times, the hot new thing is so great that it’s worth the downsides. For me, Rails is the best example of that. It’s pretty old and boring today–this is your grampa’s framework!–but at one point it was the hot new thing. After trying it out on some side projects, I found that I could roll out new apps so rapidly compared to the other options available at the time that it was worth the risks.
How to handle the hotness (if you must)
So, you’ve decided to use something new and relatively untried in production. Here are a few strategies to limit the damage.
- Roll it out slowly.
Don’t rewrite the whole world overnight. Try to convert your infrastructure one piece at a time, and have a rollback plan for each step. This is a good approach for any rollout, but it goes double for something new.
- Standardize your usage of the new technology.
You don’t want to explore the entire space of possible problems, so provide guidance to fellow engineers and try to funnel them into a single way of doing things. Duplicate emerging standards, if possible. Consider providing tooling or client libraries to help out.
- Use new tech for the purpose it was meant for.
I’ll be honest with you: I have made this mistake. I once tried to replace some rollup tables in a relational database with Lucerne indices. It seemed like a good idea at the time, but I can show you the scars. If you use a new technology in a way the creators didn’t expect, and that nobody else is doing, you are going to have a bad time.
- Don’t roll out more than one exciting thing at a time.
If you already know the quirks and problems of your existing technologies, it will be a lot easier to narrow down the issues with your new one. Moreover, exciting technology does better in a boring environment, because its assumptions about the rest of the stack will hold true.
Non-standard technology choices are enormously expensive. They have a huge up-front cost in time and effort, and then they put a multiplier on everything you ever do with that system. Experience, familiarity, and homegrown infrastructure can all help, but those things are not free. You will literally be paying for that decision for the lifetime of the system.
Some people may be disappointed not to use the fun new toys, but I don’t see it that way. I don’t want to hold you back, I want to set you free. I don’t want you hung up for weeks on compatibility issues, or waiting for custom hand-hacked infrastructure. I want your crazy, beautiful, insane, awesome ideas to be out there making the world a better place. I want them to actually work. I want the code to just flow through your fingers out into the world, but it will only do that if it’s built on the most reliable, well understood, and stable foundation possible. Your crazy ideas will work better if you build them on boring technology.
So, if you’re going to go with a non-standard technology choice, make damn sure it’s worth it. Or better yet, just go with the standard. Embrace boring. Be awesome!
Interested in seeing some additional data and examples of specific technologies decisions? Watch the full talk I gave on this subject at Cascadia Ruby 2014 below.