For the 2012 US Presidential election, the Obama for America (OFA) organization had only one goal: the re-election of President Barack Obama. The group was responsible for raising campaign contributions, registering new voters, mobilizing supporters, and persuading Americans to support the campaign.
The organization benefited from a robust web presence, raising approximately $670 million in online donations between April 2011 and November 2012 – a remarkable feat by any measure, but especially when you consider that the average donation was $55. After Obama’s inauguration in January 2013, OFA transitioned into the nonprofit advocacy group, Organizing for Action, and changed its focus to supporting the President’s executive agenda for the remainder of his time in office.
Scaling the OFA Platform
While political campaigns will always require a great deal of face-to-face contact, the OFA team understood that high-touch interactions could be helped by engaging with voters online. All of the developers on the OFA team had experience in the world of startups, but none were prepared for the intensity and scale of the Presidential campaign. “Right after the President gave his convention speech, we were doing about two million dollars an hour,” explains Nick Leeper, Tech Lead for Finance and Fundraising at OFA. “When you’re accustomed to hourly donations in the $10,000 to $20,000 range, that’s an incredible spike. You’re basically going from a thousand people using your app to 500,000 in a matter of minutes.”
“On a campaign like this, the traffic patterns are crazy,” adds Scott VanDenPlas, Director of DevOps. “One send through every channel hits 60 million people – and that’s just the first and second layer of folks who hear about it.”
Insourcing the Tech Team
Traditionally, political campaigns have outsourced their operations, which often led to a communications gap between campaign operations and the tech teams. The OFA team decided against this approach early on. Instead, they chose to bring much of the development in-house using their own technical team.
“These days I don’t think you can run the technical side of a campaign through an IT vendor,” says Jason Kunesh, OFA’s Director of User Experience. “It’s become such a rapidly moving, dynamic environment that you don’t have time to put together an RFP when you need something or when something goes wrong. You just need to push out one app after another, scrapping the apps that don’t work and optimizing the ones that do. Consider this: over the course of 18 months our internal team produced more than 200 applications. There’s just no ways you do that with a third party.”
“Prior to 2012, the national campaign operation was dealing with many silos of data from vendors and individual states, so it was difficult to standardize that data for targeting large groups of votes across the country,” adds Leeper. “In the four years since Obama’s first run, the size and influence of social networks, and in order to achieve outreach on a truly national scale, we needed to centralize our IT operations and manage all that data ourselves. Otherwise we would never have a total picture of what was happening on the ground.”
Building the Narwhal Project
Due to significant innovations, most of the technology introduced in the 2008 campaign was unusable. This meant that the OFA team essentially began from scratch as a greenfield operation. They began building a platform known internally as ‘Narwhal,’ which eventually became the services background to 18 different applications. It was designed to integrate data across a wide range of apps and the OFA team built it to be maximally redundant. If any part of the system failed, Narwhal would ensure essential functions would still be up and running.
“We kept saying that we were building an airplane in mid-flight,” jokes Ryan Kolak, Team Lead for the Narwhal Project. “While my team made Narwhal a reality, the rest of the team was building apps against APIs and platforms that didn’t exist yet. The only way to make it all work was to observe solid fundamentals, making every part of the system fault-tolerant. Without that level of discipline, we would’ve had chaos on our hands.”
One of the team’s first (and best) decisions was to run all of their applications on Amazon Web Services. “Instead of trying to run this huge infrastructure ourselves, we were able to focus on optimizing it,” says Kolak. “Our first mega-rush of traffic happened in May when Obama announced his support for gay marriage. Almost immediately, we were pushing four gigabytes per second and all of our vendors’ traffic failed over to our infrastructure. That was also the day when all of our pain points were exposed. That was also around the time we began using New Relic, which helped us identify those pain points much more quickly and much more proactively.”
Succeeding with New Relic
When the OFA team first deployed New Relic, they were able to see results in just minutes. “The first time we turned New Relic onto Narwhal, we realized that a 10-row table containing all our consumer keys was causing a big slowdown in queries,” explains Kolak. “Within 30 minutes, we’d made a patch and cached those keys. It was maybe two lines of code, and making that single change reduced 160 millisecond requests by 100 milliseconds. That was an 80 percent increase in Narwhal performance in our first hour of using this software.”
The following days brought even more dramatic improvements. “When you’re building software in an agile environment, it’s easy to overlook simple fixes during the initial build,” comments Kunesh. “New Relic pointed us to some of the blind spots that had emerged during our first few months of work. And within a few days, we’d increased our response times across the board by 90 percent.”
Improving on Tradiion
For all the OFA team’s success in solving persistent problems, they insist that specific solutions aren’t as important as the team’s overall approach. “Narwhal was built to solve very unique problems,” explains Kunesh. “Trying to emulate what we did with that platform would be a mistake. Instead, I would encourage campaigns to hire tech teams with a wide range of experience. None of us are language zealots; we’re all polyglots. And we understand not just technology, but how to map that technology over to other disciplines like business or design. It goes back to the old silo strategy, opting instead for a more centralized, holistic approach. Having people with a broad perspective, sharp folks with their feet in more than one world, puts you in a much better position to build around failure scenarios and create healable applications.”
With tools like New Relic at their disposal, the OFA tech team focused more of its energy on iteration and less on reinvention. Due to their multi-year cycles, political campaigns are the perfect place for truly agile development. “If people look back in a year or two, I really hope they’re not using the code we wrote because it’s going to stagnate,” says Kunesh. “Things change. Vendors change. Technologies change. Apps come and go. That’s just the way things are. Personally, I’ll be very happy if people look back at our work and say, ‘You put a really smart team together. You put them in the same room and you gave them really tough problems to solve. And you inspired the next campaign to tackle even bigger problems on their own terms and in their own way.”
Read the Full Case Study
To find out how New Relic helped the OFA team improve their app performance by 90%, read the full OFA case study.