Nate Silver's 3 Secrets to Handling Big Data
Watch the video of Nate Silver’s entire Summit presentation, including audience Q&A.
If there’s such a thing as a “data rockstar”, Nate Silver would be one of the names that comes to mind. After all, Silver successfully predicted the outcome of voting in 49 of 50 states in the 2008 U.S. Presidential election, was named one of the World’s 100 Most Influential People in 2009 by Time Magazine, and published a New York Times Best Seller, The Signal and the Noise, in 2012.
In fact, Silver acknowledged his notoriety when opening his special presentation to a packed house at the Microsoft Data Insights Summit on March 23, 2016. During a period in 2013, he noted, “Nate Silver” was used as a search term over twice as often as U.S. Vice President Joe Biden – although both were many multitudes short of pop sensation “Justin Bieber”. Hype is always a matter of scale.
The concept of hype applies to big data as well. Like any new technology, big data has gone through a hype phase that inflated expectations but, Silver argues, it’s currently in a “plateau of productivity”. People are no longer satisfied with just the promise of big data. Now they want to see proof of why it matters and what we can do with it.
In his Summit presentation, Silver pointed out three problems that data analysts should consider when approaching their work. First is the tendency to connect too much data in a way that isn’t systematic. Additionally, with more and more data being produced all the time, there is a serious signal-to-noise ratio issue for any data wrangler. Finally, there can be a tendency to let algorithms do the “thinking” when in fact our own gut feeling is almost always more important.
Silver also had three suggestions to overcome these problems. He emphasized the need for practice and always making sure that both your technology and your scientific theory is up to date. Understanding your own bias is important for accurate insights. And finally, never ignore the importance of trial and error and taking your mistakes from this project and avoiding them on the next one.
Silver closed out his Data Insight Summit presentation with a quotation from Piet Hein, Danish scientist and poet:
The road to wisdom?
Well, it’s plain
and simple to express.
Err and err and err again,
but less and less and less.
You can watch Silver’s list of three problems facing big data and three suggestions to avoid them, along with audience Q&A, in our video from the Summit.