ABS hobbles census data downloaders

 

DVDs charged at $250 a pop.

The Australian Bureau of Statistics has released the latest census data for free under a Creative Commons license but appears to be steering people towards a $250 mailed out DVD rather than making it easy to download the information directly over the internet.

Programmer and freelance journalist Grahame Bowland who first noticed it, said the government agency is going to great lengths to discourage people from downloading the files directly by dint of  a convoluted site layout and Javascript functions that obfuscate file paths.

Bowland discovered how difficult it was to download the data packs for free when he tried to do it, with the ABS forcing him to jump through several hoops before accessing the data.

First, people have to register on a hard-to-find page, Bowland said, after which they are redirected to another page with a big matrix of data packs.

ABS census datapacks download page

"You have to click to download each pack individually, and they've set the site up deliberately to make it difficult to use a browser plugin to download everything that is contained on the released DVD image," Bowland told iTNews.

Javascript that obfuscates the file paths has been added to the site by the ABS developers, including comments. Bowland provided the below commented code sample directly from the agency site to iTNews:

// Function: guidGenerator
// Description:returns a pseudo-random GUID
//This is appended to a url for 2 reasons
//1. to make the URL unique, so that the browser always gets it and doesn't use a cached version
//2. to make a URL look like its got a unique key, in a naive attempt to fool a not-so-wily hacker
//into thinking they can't download a datapack directly if they know the URL pattern, because they
//need a unique key.
 

"The ABS is trying to obfuscate paths to make it hard for people to bulk download the data, and labelling people who want to 'hackers'," Bowland said.

Bowland said on closer inspection of the code it appears you don't actually have to register at all to get the data packs, as you can use a Javascript code snippet to generate the URLs of all the data packs.

To prevent this, the ABS Javascript coders have documented what they call a "pathetic attempt" at stopping the direct data pack downloads by using a randomly generated number appended to the URLs so as to make them appear as if a complex key is required as per below:

// Function: getZip
// Parameters:fileName - the file to be downloaded.
//There are basically 2 formats, normal DataPacks and boundary files. For example:
//* 2010_BCP_SA1_for_Vic_short-header.zip
//* 2010_SA4_POW_shape.zip  or without the POW eg. 2010_SA4_shape.zip
// Descrition:
// This function is ultimately fired when a user clicks on a download link on the DataPacks download page.
// It does 4 things:
//
// Step 1. Get some dynamic values from the page. These were substituted when the page was created.
//'dpserver' is the full domino path, as seen in the dominoserver.properties value DataPacks.DominoServerExt
//  eg. http://www.censusdata.idev.abs.gov.au/CensusOutput/copsubdatapacks.nsf/All%20docs%20by%20catNo,
//      Also, generate a random number, which we append to the URL, to make it appear as if a complex
//key is required. This is a pathetic attempt to discourage someone from downloading the ZIPs
//directly (ie. without having to login), if they deduce the URL pattern.
//It's also used to make every URL unique, so that the browser always sends the request to the server,
//(ie. doesn't use its cached version), because we want to know about every click.

Despite having got this far, Bowland noted that the some additional geometry files for DVD 3 couldn't be found on the ABS website, so he decided to stump up $250 for all the releases to be mailed to him.

According to Bowland, the high cost set to post out an optical disc means "in reality, they're subsidising internal admin roles by selling DVDs".

For those who don't wish to tangle with obfuscating Javascript or pay hefty charges for DVDs, Bowland has made the census data available for download via Bittorrent on his website.

ABS responds

A spokesperson for the ABS said the $250 charge for the DVDs was to "recover administration costs" but pointed out that this was the first time its census data had been made free and available to everyone via its website.

According to the spokesperson, the ABS has worked hard to reduce the costs since 2006, when similar datapacks cost $805.

As for the convoluted download site layout with registration and obfuscated file paths, the spokesperson said there was room for improvement.

"The ABS is constantly looking at ways it can simplify the website and enhance the user experience," iTnews was told via email.

"We will shortly be conducting a review of all census products and services and will engage users of census data to better understand their needs," the spokesperson added.

Copyright © iTnews.com.au . All rights reserved.


ABS hobbles census data downloaders
 
 
 
Top Stories
Frugality as a service: the Amazon story
Behind the scenes, Amazon Web Services is one lean machine.
 
Negotiating with the cloud email megavendors
[Blog post] Lessons from Woolworths’ mammoth migration.
 
Qld govt to move up to 149k staff onto Office 365
Australia's largest deployment, outside of the universities.
 
 
Sign up to receive iTnews email bulletins
   FOLLOW US...

Latest VideosSee all videos »

The great data centre opportunity on Australia's doorstep
The great data centre opportunity on Australia's doorstep
Scott Noteboom, CEO of LitBit speaking at The Australian Data Centre Strategy Summit 2014 in the Gold Coast, Queensland, Australia. http://bit.ly/1qpxVfV Scott Noteboom is a data centre engineer who led builds for Apple and Yahoo in the earliest days of the cloud, and who now eyes Asia as the next big opportunity. Read more: http://www.itnews.com.au/News/372482,how-do-we-serve-three-billion-new-internet-users.aspx#ixzz2yNLmMG5C
Interview: Karl Maftoum, CIO, ACMA
Interview: Karl Maftoum, CIO, ACMA
To COTS or not to COTS? iTnews asks Karl Maftoum, CIO of the ACMA, at the CIO Strategy Summit.
Susan Sly: What is the Role of the CIO?
Susan Sly: What is the Role of the CIO?
AEMO chief information officer Susan Sly calls for more collaboration among Australia's technology leaders at the CIO Strategy Summit.
Meet the 2014 Finance CIO of the Year
Meet the 2014 Finance CIO of the Year
Credit Union Australia's David Gee awarded Finance CIO of the Year at the iTnews Benchmark Awards.
Meet the 2014 Retail CIO of the Year
Meet the 2014 Retail CIO of the Year
Damon Rees named Retail CIO of the Year at the iTnews Benchmark Awards for his work at Woolworths.
Robyn Elliott named the 2014 Utilities CIO of the Year
Robyn Elliott named the 2014 Utilities CIO of the Year
Acting Foxtel CIO David Marks accepts an iTnews Benchmark Award on behalf of Robyn Elliott.
Meet the 2014 Industrial CIO of the Year
Meet the 2014 Industrial CIO of the Year
Sanjay Mehta named Industrial CIO of the Year at the iTnews Benchmark Awards for his work at ConocoPhillips.
Meet the 2014 Healthcare CIO of the Year
Meet the 2014 Healthcare CIO of the Year
Greg Wells named Healthcare CIO of the Year at the iTnews Benchmark Awards for his work at NSW Health.
Meet the 2014 Education CIO of the Year
Meet the 2014 Education CIO of the Year
William Confalonieri named Healthcare CIO of the Year at the iTnews Benchmark Awards for his work at Deakin University.
Meet the 2014 Government CIO of the Year
Meet the 2014 Government CIO of the Year
David Johnson named Government CIO of the Year at the iTnews Benchmark Awards for his work at the Queensland Police Service.
Q and A: Coalition Broadband Policy
Q and A: Coalition Broadband Policy
Malcolm Turnbull and Tony Abbott discuss the Coalition's broadband policy with the press.
AFP scalps hacker 'leader' inside Australia's IT ranks.
AFP scalps hacker 'leader' inside Australia's IT ranks.
The Australian Federal Police have arrested a Sydney-based IT security professional for hacking a government website.
NBN Petition Delivered To Turnbull's Office
NBN Petition Delivered To Turnbull's Office
UTS CIO: IT teams of the future
UTS CIO: IT teams of the future
UTS CIO Chrissy Burns talks data.
New UTS Building: the IT within
New UTS Building: the IT within
The IT behind tomorrow's universities.
iTnews' NBN Panel
iTnews' NBN Panel
Is your enterprise NBN-ready?
Introducing iTnews Labs
Introducing iTnews Labs
See a timelapse of the iTnews labs being unboxed, set up and switched on! iTnews will produce independent testing of the latest enterprise software to hit the market after installing a purpose-built test lab in Sydney. Watch the installation of two DL380p servers, two HP StoreVirtual 4330 storage arrays and two HP ProCurve 2920 switches.
The True Cost of BYOD
The True Cost of BYOD
iTnews' Brett Winterford gives attendees of the first 'Touch Tomorrow' event in Brisbane a brief look at his research into enterprise mobility. What are the use cases and how can they be quantified? What price should you expect to pay for securing mobile access to corporate applications? What's coming around the corner?
Ghost clouds
Ghost clouds
ACMA chair Chris Chapman says there is uncertainty over whether certain classes of cloud service providers are caught by regulations.
Was the Snowden leak inevitable?
Was the Snowden leak inevitable?
Privacy experts David Vaile (UNSW Cyberspace Law and Policy Centre) and Craig Scroggie (CEO, NextDC) claim they were not surprised by the Snowden leaks about the NSA's PRISM program.
Latest Comments
Polls
Which bank is most likely to suffer an RBS-style meltdown?





   |   View results
ANZ
  21%
 
Bankwest
  9%
 
CommBank
  11%
 
National Australia Bank
  17%
 
Suncorp
  24%
 
Westpac
  19%
TOTAL VOTES: 1458

Vote