For Bill Sammons, head of content enrichment at Dow Jones & Company, “scale” isn’t just a buzzword, it’s an ever-present reality that drives decision-making in all sorts of ways, from updating the technology stack to setting long-term goals.
The implications of scale became clear in Bill’s engrossing and informative presentation at New Relic’s FutureStack16 Tour: New York event this month. In Creating Modern Metadata Systems with New Relic, he described in detail the challenges and approaches of Dow Jones’ technology stack transformation, leveraging New Relic Insights to transition the company’s classification engine.
This was far from a trivial task, as just 18 months ago the company was relying on a monolithic architecture built around “stale” turn-of-the-century technologies to apply new metadata to 30-year archive of 1.5 billion articles across three taxonomies, thousands of nodes, and seven languages. The company needed to hit “reset,” Bill said, to be ready to add up to 1 billion new articles to its content ingestion and enrichment pipeline every month! (The key to the value of that archive, he noted, is to quickly add metadata that can differentiate an article about some salesman in Iowa named Steve Jobs from an article about that Steve Jobs, for example.)
Watch the video below to see how Bill’s team built a new stack based on modern cloud technologies, moved from a waterfall development model to agile and scrum, and leveraged New Relic Insights to make it easy to check the performance of the complex new system with just a few dashboards. Dow Jones updates the system every week with a new config, but really worried about performance, Bill said. With Insights, he can see how things are changing with every new release: “As long as I can see with my eye that things are staying roughly the same, then we’re good!”
Dow Jones learned many important lessons along the way, but these five critical points stuck in my mind:
1. Big may not be big enough
Dow Jones legacy system was ingesting and enriching 200 million new articles a month, from some 1,500 different sources. That’s a lot, obviously, but while the system was big, Bill said, it wasn’t big enough.
2. NRQL is “SQL for Managers”
“It’s great,” Bill said of the New Relic Query Language used to work with Insights. “It’s actually very easy to do complex things.” He writes NRQL code himself, and “if a manager can do it, it’s not that hard.” Just as important, Insights helps his team be “ready to answer questions not yet asked.” Even when complex questions come up from his team’s business partners, he said, “six hours later we have a dashboard, [and] they have their answers.”
3. If you’re not using Slack, start using Slack
Communication is key, Bill said, and credited next-generation chat tools to speeding the development process. “If you’re not using Slack, start using Slack,” Bill said. “My email used to be full, now my email doesn’t have anything in it. It’s great!”
4. AWS rocks
“I can’t say enough about AWS,” Bill gushed. “The richness of the tech base is great”—there are a lot of tools there that turbocharge your environment. “We do leverage CloudFormation a lot,” he added, noting that it’s “absolutely amazing, liberating, to be able to do new and interesting things.”
5. “If someone says you can’t do it … you can!”
With a cloud-based technology stack and a modern approach to development, Bill said, “impossible tasks are now possible.… We can do things we always wanted to do but could never afford to do or never conceive how to do. It’s not only doable, but easily doable. They’re not only easily doable, they’re cost effective!”
To learn more, watch the video of Bill’s 30-minute presentation below:
And to make sure you don’t miss the details, we’ve embedded his entire presentation deck below:
Join us for FutureStack in San Francisco
Note: Event dates, participants, and topics are subject to change without notice.