As many data nerds know, Nate Silver made his name by doing better than almost any other pundit in correctly predicting the results of the 2008 and 2012 Presidential elections. Well, in 2016, he wasn’t as wrong as most statisticians, giving Clinton only a little more than 70% chance of winning (not far from the Trump campaign’s own predictions), while others gave her odds of up to 99%.
So in the wake of an election season that stunned most observers, what did the founder and editor-in-chief of FiveThirtyEight.com have to say to an audience of software and analytics professionals in his keynote presentation yesterday at FutureStack16 San Francisco?
Plenty, as it turns out.
To help explain the surprise election results, Silver focused on the relatively large number of undecided voters, and the fact that results from the swing states were correlated, not independent.
Beyond the election
But that was only a small part of his fascinating presentation. Silver talked about the strengths, weaknesses, and persistent misunderstandings surrounding statistics and analytics, and addressed the differences and similarities between the work he does and what New Relic customers do. (Political polling is “moderately complex,” he said, but less so than “what the people in this room do.”) And, most important, he offered some real-world tips on how best to use “big data” to make more accurate predictions.
First off, Silver addressed the “wall of infamy” surrounding critical events that were not properly predicted, from the housing bubble and economic collapse in 2008 to the 9/11 terrorist attacks to the unexpectedly massive damage caused by the Fukushima earthquake and tsunami. Just as important, he noted high-profile false positives, including predictions of massive outbreaks of bird flu and SARS, neither of which came to pass. Citing the computer revolution of the 1980s, Silver noted that it often takes longer than expected for new technologies like big data analytics to prove their worth. Many people give up at that point, he said, “but the companies out in front tend to really benefit” when those technologies do finally come of age.
To explain the difficulties involved in making real-world predictions, Silver shared three key problems that should resonate in many software shops:
Problem #1: The more data you have, the more room there is for interpretation. Sure, more data is a good thing, but it also makes the situation more complex. In the old days, Silver said, there were only 3-4 election polls a week; now there are dozens of polls and aggregators, and the polling process has become politically charged. “We’re still at a very infantile place in interpreting polls,” Silver said.
Problem #2: The signal-to-noise ratio. Playing off the title of his book, his point was that as data sets get larger, they undergo an exponential increase in complexity. Increasing the size of a meeting from 5 people to 10 people, for example, far more than doubles the possible complications and obstacles. The noise grows faster than the signal. The problem, he explained, is that larger data sets vastly increase the chance of false positives, of correlations without causation. “And betting on correlations when you’re not sure of the causation … that’s a very dangerous bet.”
Problem #3: Feature or bug? Too often, he said, the role of common sense and gut instinct in predictions is misunderstood. If you apply “common sense” too early, you risk overriding the value of the data and the model through observer bias and other factors. When a company believes it has discovered an opportunity its competitors don’t see, for example, “that’s a bold claim … your competitors are likely to be almost as smart as you,” so you’re claiming there’s a bug in your model, not in your insight, which may not be true. By applying your insight at the right time, you can validate the model and the corrections you make are more likely to be supported by the data.
Silver also shared some suggestions on how to avoid those problems and make better predictions.
Suggestion #1: Think Probabilistically. Predictors tend to understate uncertainty, Silver said, but predictions that convey uncertainty are usually better forecasts. It can be confusing to deal with probabilities of binary outcomes, but Donald Trump is no less president because the election was close. He added that visual aids—such as storm track maps—can often help people perceive uncertainty more clearly. “A picture can do a lot of good.”
Suggestion #2: Know Where You’re Coming From. It’s important to understand your biases, Silver said, but most people don’t. For instance, when male and female candidates with identical qualifications apply for a job, tests show that people who say they have no gender bias actually show more gender bias.
“Avoid group think,” Silver warned, “Diversity of perspectives can mitigate risk.” The right kind of experience can help. Silver suggested that one reason FiveThirtyEight.com was “less wrong” about the election than its competitors is because his outfit has more hands-on experience than many others. Predictors with experience making predictions with money on the line, such as poker players or sports analysts, tend to do better than academics who haven’t learned the hard way that they need to challenge their assumptions.
Suggestion #3: Try, and Err. Learning and refining your models is critical to improving your predictions, Silver said, but that works best in a data-rich environment. Weather forecasting has improved dramatically, he said, because meteorologists get a lot of practice: “If you have a bad model, you get it beat out of you pretty fast. Presidential election forecasters only get one trial every four years.”
The key, Silver concluded, is to never be satisfied with where you stood before. While it’s easy for your models to be overconfident, “real data is a reality check.”
Nate Silver photos: © Andrew Weeks Photography