Nate Silver proves data analysis can pick presidents

By on
Nate Silver proves data analysis can pick presidents
iTnews columnist Justin Warren.

Opinion: A big tick for big data?

The 2012 US Election was a triumph of facts, statistics and analysis over wishful thinking.

Nate Silver managed to repeat previous successes in combining aggregated poll data and a sophisticated statistical model to accurately predict the outcomes of the Presidential race.

Similarly, the Princeton Election Consortium has churned out accurate predictions, but hasn't made quite the same contribution to popular sports as Silver.

There has been significant resistance to Silver's predictions in the past couple of weeks, with various commentators, frequently conservatives, calling him "partisan" and an "ideologue".

"Nobody knows anything. Everyone's guessing," commented Peggy Noonan of The Wall Street Journal.

Silver, to his credit, has frequently demurred.

"I'm sure that I have a lot riding on the outcome. I'm also sure I'll get too much credit if the prediction is right and too much blame if it is wrong," he said.

It appears some credit is due. This graphic shows Silver's predictions side-by-side with the actual results, and they're a near perfect match.

Silver, and his reliance on empirical evidence, has been vindicated. Perhaps Noonan and her ilk may yet concede that there are some people who do know things, and aren't guessing.

Silver's success will prove fortunate for the ABC's Antony Green, Crikey's Possum Comitatus and Greg Jericho.

Even more likely, expect to see a lot of news items coming out in praise of "big data" and how great it is for this sort of thing, particularly from vendors who sell the gear.

They're only partly right.

The data here isn't the star. The analysis is.

Silver, and those like him, have access to the same data as everyone else. This isn't an advantage borne of access to a gigantic dataset that no one else has. This is an ability to use that data, to ask the right questions, and correctly interpret the results.

Expensive technology can provide you with the tools, and the raw fire-power, to crunch numbers quickly, but it can't help you to figure out what data is important, or how to decide.

For that, you need to understand how analysis works, particularly statistics. This is a realm that most people find deathly boring when they're exposed to it in high school, and likely never touch again. And as Kahneman and Tversky discovered in the 1970s, humans are naturally poor at dealing with probability.

Just this week, Australians spent almost $100 million on the Melbourne Cup, and millions more on a $100 million jackpot lotto draw. Most will walk away with nothing.

To get the real value out of big data, you need to understand statistics, and how to use the data well. An estimate is not a guess. Confidence intervals and R-squared figures provide real insight, both of what you do know, and, importantly, what you do not.

Good analysis provides a way of testing your assumptions, of refining your thinking, and guiding you to a better and deeper understanding of what's actually going on.

Relying on gut instinct is as doomed to failure as your ability to win first division lotto. 

"A lottery is a tax on those who are bad at math." — Ambrose Bierce

Poorly considered investments in big data may be the same.

Copyright © iTnews.com.au . All rights reserved.
Tags:

Most Read Articles

You must be a registered member of iTnews to post a comment.
| Register

Poll

How should the costs of Australia's piracy scheme be split?
Rights holders should foot the whole bill
50/50
ISPs should foot the whole bill
Government should chip in a bit
Other
Flash is heading towards its grave, and that's...
Great! Good riddance
Sad! Flash had some good qualities
Irrelevant. I don't care
What's Flash?
View poll archive

Whitepapers from our sponsors

What will the stadium of the future look like?
What will the stadium of the future look like?
New technology adoption is pushing enterprise networks to breaking point
New technology adoption is pushing enterprise networks to breaking point
Gartner names IBM a 'Leader' for Disaster Recovery as a Service
Gartner names IBM a 'Leader' for Disaster Recovery as a Service
The next era of business continuity: Are you ready for an always-on world?
The next era of business continuity: Are you ready for an always-on world?

Log In

Username:
Password:
|  Forgot your password?