Nate Silver proves data analysis can pick presidents

 

Opinion: A big tick for big data?

The 2012 US Election was a triumph of facts, statistics and analysis over wishful thinking.

Nate Silver managed to repeat previous successes in combining aggregated poll data and a sophisticated statistical model to accurately predict the outcomes of the Presidential race.

Similarly, the Princeton Election Consortium has churned out accurate predictions, but hasn't made quite the same contribution to popular sports as Silver.

There has been significant resistance to Silver's predictions in the past couple of weeks, with various commentators, frequently conservatives, calling him "partisan" and an "ideologue".

"Nobody knows anything. Everyone's guessing," commented Peggy Noonan of The Wall Street Journal.

Silver, to his credit, has frequently demurred.

"I'm sure that I have a lot riding on the outcome. I'm also sure I'll get too much credit if the prediction is right and too much blame if it is wrong," he said.

It appears some credit is due. This graphic shows Silver's predictions side-by-side with the actual results, and they're a near perfect match.

Silver, and his reliance on empirical evidence, has been vindicated. Perhaps Noonan and her ilk may yet concede that there are some people who do know things, and aren't guessing.

Silver's success will prove fortunate for the ABC's Antony Green, Crikey's Possum Comitatus and Greg Jericho.

Even more likely, expect to see a lot of news items coming out in praise of "big data" and how great it is for this sort of thing, particularly from vendors who sell the gear.

They're only partly right.

The data here isn't the star. The analysis is.

Silver, and those like him, have access to the same data as everyone else. This isn't an advantage borne of access to a gigantic dataset that no one else has. This is an ability to use that data, to ask the right questions, and correctly interpret the results.

Expensive technology can provide you with the tools, and the raw fire-power, to crunch numbers quickly, but it can't help you to figure out what data is important, or how to decide.

For that, you need to understand how analysis works, particularly statistics. This is a realm that most people find deathly boring when they're exposed to it in high school, and likely never touch again. And as Kahneman and Tversky discovered in the 1970s, humans are naturally poor at dealing with probability.

Just this week, Australians spent almost $100 million on the Melbourne Cup, and millions more on a $100 million jackpot lotto draw. Most will walk away with nothing.

To get the real value out of big data, you need to understand statistics, and how to use the data well. An estimate is not a guess. Confidence intervals and R-squared figures provide real insight, both of what you do know, and, importantly, what you do not.

Good analysis provides a way of testing your assumptions, of refining your thinking, and guiding you to a better and deeper understanding of what's actually going on.

Relying on gut instinct is as doomed to failure as your ability to win first division lotto. 

"A lottery is a tax on those who are bad at math." — Ambrose Bierce

Poorly considered investments in big data may be the same.

Copyright © iTnews.com.au . All rights reserved.


Nate Silver proves data analysis can pick presidents
iTnews columnist Justin Warren.
 
 
 
Top Stories
Meet FABACUS, Westpac's first computer
GE225 operators celebrate gold anniversary.
 
NSW Govt gets ready to throw out the floppy disks
[Opinion] Dominic Perrottet says its time for government to catch up.
 
iiNet facing new copyright battle with Hollywood
Fighting to protect customer details.
 
 
iTnews columnist Justin Warren.
Sign up to receive iTnews email bulletins
   FOLLOW US...
Latest Comments
Polls
In which area is your IT shop hiring the most staff?




   |   View results
IT security and risk
  26%
 
Sourcing and strategy
  12%
 
IT infrastructure (servers, storage, networking)
  21%
 
End user computing (desktops, mobiles, apps)
  15%
 
Software development
  26%
TOTAL VOTES: 339

Vote
Would your InfoSec team be prepared to share threat data with the Australian Government?

   |   View results
Yes
  58%
 
No
  42%
TOTAL VOTES: 143

Vote