Nate Silver proves data analysis can pick presidents

 

Opinion: A big tick for big data?

The 2012 US Election was a triumph of facts, statistics and analysis over wishful thinking.

Nate Silver managed to repeat previous successes in combining aggregated poll data and a sophisticated statistical model to accurately predict the outcomes of the Presidential race.

Similarly, the Princeton Election Consortium has churned out accurate predictions, but hasn't made quite the same contribution to popular sports as Silver.

There has been significant resistance to Silver's predictions in the past couple of weeks, with various commentators, frequently conservatives, calling him "partisan" and an "ideologue".

"Nobody knows anything. Everyone's guessing," commented Peggy Noonan of The Wall Street Journal.

Silver, to his credit, has frequently demurred.

"I'm sure that I have a lot riding on the outcome. I'm also sure I'll get too much credit if the prediction is right and too much blame if it is wrong," he said.

It appears some credit is due. This graphic shows Silver's predictions side-by-side with the actual results, and they're a near perfect match.

Silver, and his reliance on empirical evidence, has been vindicated. Perhaps Noonan and her ilk may yet concede that there are some people who do know things, and aren't guessing.

Silver's success will prove fortunate for the ABC's Antony Green, Crikey's Possum Comitatus and Greg Jericho.

Even more likely, expect to see a lot of news items coming out in praise of "big data" and how great it is for this sort of thing, particularly from vendors who sell the gear.

They're only partly right.

The data here isn't the star. The analysis is.

Silver, and those like him, have access to the same data as everyone else. This isn't an advantage borne of access to a gigantic dataset that no one else has. This is an ability to use that data, to ask the right questions, and correctly interpret the results.

Expensive technology can provide you with the tools, and the raw fire-power, to crunch numbers quickly, but it can't help you to figure out what data is important, or how to decide.

For that, you need to understand how analysis works, particularly statistics. This is a realm that most people find deathly boring when they're exposed to it in high school, and likely never touch again. And as Kahneman and Tversky discovered in the 1970s, humans are naturally poor at dealing with probability.

Just this week, Australians spent almost $100 million on the Melbourne Cup, and millions more on a $100 million jackpot lotto draw. Most will walk away with nothing.

To get the real value out of big data, you need to understand statistics, and how to use the data well. An estimate is not a guess. Confidence intervals and R-squared figures provide real insight, both of what you do know, and, importantly, what you do not.

Good analysis provides a way of testing your assumptions, of refining your thinking, and guiding you to a better and deeper understanding of what's actually going on.

Relying on gut instinct is as doomed to failure as your ability to win first division lotto. 

"A lottery is a tax on those who are bad at math." — Ambrose Bierce

Poorly considered investments in big data may be the same.

Copyright © iTnews.com.au . All rights reserved.


Nate Silver proves data analysis can pick presidents
iTnews columnist Justin Warren.
 
 
 
Top Stories
Frugality as a service: the Amazon story
Behind the scenes, Amazon Web Services is one lean machine.
 
Negotiating with the cloud email megavendors
[Blog post] Lessons from Woolworths’ mammoth migration.
 
Qld govt to move up to 149k staff onto Office 365
Australia's largest deployment, outside of the universities.
 
 
iTnews columnist Justin Warren.
Sign up to receive iTnews email bulletins
   FOLLOW US...

Latest VideosSee all videos »

The great data centre opportunity on Australia's doorstep
The great data centre opportunity on Australia's doorstep
Scott Noteboom, CEO of LitBit speaking at The Australian Data Centre Strategy Summit 2014 in the Gold Coast, Queensland, Australia. http://bit.ly/1qpxVfV Scott Noteboom is a data centre engineer who led builds for Apple and Yahoo in the earliest days of the cloud, and who now eyes Asia as the next big opportunity. Read more: http://www.itnews.com.au/News/372482,how-do-we-serve-three-billion-new-internet-users.aspx#ixzz2yNLmMG5C
Interview: Karl Maftoum, CIO, ACMA
Interview: Karl Maftoum, CIO, ACMA
To COTS or not to COTS? iTnews asks Karl Maftoum, CIO of the ACMA, at the CIO Strategy Summit.
Susan Sly: What is the Role of the CIO?
Susan Sly: What is the Role of the CIO?
AEMO chief information officer Susan Sly calls for more collaboration among Australia's technology leaders at the CIO Strategy Summit.
Meet the 2014 Finance CIO of the Year
Meet the 2014 Finance CIO of the Year
Credit Union Australia's David Gee awarded Finance CIO of the Year at the iTnews Benchmark Awards.
Meet the 2014 Retail CIO of the Year
Meet the 2014 Retail CIO of the Year
Damon Rees named Retail CIO of the Year at the iTnews Benchmark Awards for his work at Woolworths.
Robyn Elliott named the 2014 Utilities CIO of the Year
Robyn Elliott named the 2014 Utilities CIO of the Year
Acting Foxtel CIO David Marks accepts an iTnews Benchmark Award on behalf of Robyn Elliott.
Meet the 2014 Industrial CIO of the Year
Meet the 2014 Industrial CIO of the Year
Sanjay Mehta named Industrial CIO of the Year at the iTnews Benchmark Awards for his work at ConocoPhillips.
Meet the 2014 Healthcare CIO of the Year
Meet the 2014 Healthcare CIO of the Year
Greg Wells named Healthcare CIO of the Year at the iTnews Benchmark Awards for his work at NSW Health.
Meet the 2014 Education CIO of the Year
Meet the 2014 Education CIO of the Year
William Confalonieri named Healthcare CIO of the Year at the iTnews Benchmark Awards for his work at Deakin University.
Meet the 2014 Government CIO of the Year
Meet the 2014 Government CIO of the Year
David Johnson named Government CIO of the Year at the iTnews Benchmark Awards for his work at the Queensland Police Service.
Q and A: Coalition Broadband Policy
Q and A: Coalition Broadband Policy
Malcolm Turnbull and Tony Abbott discuss the Coalition's broadband policy with the press.
AFP scalps hacker 'leader' inside Australia's IT ranks.
AFP scalps hacker 'leader' inside Australia's IT ranks.
The Australian Federal Police have arrested a Sydney-based IT security professional for hacking a government website.
NBN Petition Delivered To Turnbull's Office
NBN Petition Delivered To Turnbull's Office
UTS CIO: IT teams of the future
UTS CIO: IT teams of the future
UTS CIO Chrissy Burns talks data.
New UTS Building: the IT within
New UTS Building: the IT within
The IT behind tomorrow's universities.
iTnews' NBN Panel
iTnews' NBN Panel
Is your enterprise NBN-ready?
Introducing iTnews Labs
Introducing iTnews Labs
See a timelapse of the iTnews labs being unboxed, set up and switched on! iTnews will produce independent testing of the latest enterprise software to hit the market after installing a purpose-built test lab in Sydney. Watch the installation of two DL380p servers, two HP StoreVirtual 4330 storage arrays and two HP ProCurve 2920 switches.
The True Cost of BYOD
The True Cost of BYOD
iTnews' Brett Winterford gives attendees of the first 'Touch Tomorrow' event in Brisbane a brief look at his research into enterprise mobility. What are the use cases and how can they be quantified? What price should you expect to pay for securing mobile access to corporate applications? What's coming around the corner?
Ghost clouds
Ghost clouds
ACMA chair Chris Chapman says there is uncertainty over whether certain classes of cloud service providers are caught by regulations.
Was the Snowden leak inevitable?
Was the Snowden leak inevitable?
Privacy experts David Vaile (UNSW Cyberspace Law and Policy Centre) and Craig Scroggie (CEO, NextDC) claim they were not surprised by the Snowden leaks about the NSA's PRISM program.
Latest Comments
Polls
Which bank is most likely to suffer an RBS-style meltdown?





   |   View results
ANZ
  21%
 
Bankwest
  9%
 
CommBank
  11%
 
National Australia Bank
  17%
 
Suncorp
  24%
 
Westpac
  19%
TOTAL VOTES: 1461

Vote