Best-selling author and journalist Malcolm Gladwell capped a busy and exciting first day of FutureStack: New York this week with an entertaining and thought-provoking keynote talk about the critical importance of putting data in context.
The author of cultural and intellectual touchstones like The Tipping Point, Outliers, Blink, and many others, Gladwell addressed issues of deep importance to New Relic customers who make critical business decisions based on data. He warned the assembled crowd that people tend to take the data they receive at face value, which can lead to wildly incorrect conclusions.
“It’s worth it to take a step back and be skeptical about how we use the data we get,” Gladwell said, to spend some time “interrogating the meaning of the numbers.” People often think that data is inherently objective, he pointed out, but the numbers always come with history, ideology, and implied hierarchies that impact their value and the conclusions we draw from them. “There are implicit ideologies and philosophies behind the numbers we use,” he said, and we need to understand them in order to avoid mistakes in how we use them.
The great STEM problem
Gladwell illustrated his points with multiple examples from higher education, including “data-based” rankings of U.S. universities that turn out to be almost entirely subjective and reward spending money and penalize cost-effectiveness. The point is that the numbers often don’t reflect what we really think is important.
Another example concerned the LSAT test used to determine admittance to law school. He said that limiting the test time to just three hours rewarded students who could answer easy questions quickly, not the applicants who needed more time to answer the difficult questions. But which type of person do you really want as your attorney, he wondered.
His most interesting example, though, investigated the shortage of science, technology, engineering, and math (STEM) graduates in the United States. The problem is not that too few high school graduates are interested in STEM, he said. It’s a retention problem, because about half of STEM students drop out of the programs. He drilled into the numbers to find out why.
It turns out, Gladwell explained, that factors such as college difficulty and math SAT scores don’t really predict who will make it through a STEM program. The best predictor is class rank, with the lowest scoring students at schools at all levels tending to drop out. Class rank trumps the other factors, Gladwell said, with the bottom third of students dropping out more often at institutions across the board. That means more accomplished students in the bottom of their class at elite schools are more likely to drop out than less talented students (judged by math SAT scores) at the top of their class in less prestigious institutions.
Big fish, little ponds
According to what Gladwell called Relative Deprivation, or the “Big Fish, Little Pond” theory, it seems that students measure themselves against their local peers, not the larger population. So a student with a 750 SAT score at MIT sees that she’s not at the top of her class, while another student who scored a 550 might be at the top of her class at a less demanding school. The 750 student, objectively one of the smartest kids in the world, sees herself as one of the dumbest kid in her class, Gladwell said, and decides to become a fine arts major.
Put another way, for a given student, as the average SAT scores at their school go down, their chances of ending up with a STEM degree go up! Of course, students—and their parents—hardly ever choose schools that way because all they see is the school rankings.
This is a huge issue for STEM education, of course, but it’s also an object lesson for anyone who works with data.
Gladwell challenged the FutureStack: New York audience—“people who live in the world of numbers”—to occasionally take the time to ask skeptical questions of the data they work with, to learn the stories behind the numbers, to ask “dumb” questions that challenge conventional wisdom. Otherwise, he said, “all the metrics in the world won’t make your jobs any easier.”
FutureStack: New York photos by Andres Otero