ABS prepares big data transformation

The Australian Bureau of Statistics has begun an IT transformation aimed at tackling an incoming "big data tsunami" which the agency's COO is hoping will include more personal data from citizens.

The ABS' technology infrastructure transformation will be the biggest undertaken at the department and will include serious work on renewing metadata systems and processes.

ABS chief operating officer Jonathan Palmer described the data volumes currently held by the agency as a “gentle wave lapping at the shore” ahead of the “big data tsunami,” with work already underway to enable the Bureau to deal with increasing demand.

“Right now we are mapping out a proposal for a much larger investment from government to equip us for the future,” he said.

“In the future we want to spend more time on data analysis and integration and less time on processing.

“We need smarter systems for capture and editing. We want to drive some work back out to the respondents so they can fix errors themselves. We want to use machine intelligence to do things like coding information,” he said.

The ABS already operates some of the most advanced systems in the government for the purpose of stripping personal identifiers from datasets, and it would like to share it with other federal agencies.

"We process data through our privacy preserving analytics system and make it available for members of the public to play around with, while at the same time remaining confident that no one will be able to identify individuals," he said.

"It's more sophisticated than just aggregating everything up to a high level. It's about perturbing cells and making sure that we released data in a safe way."

Would you hand over your personal data?

Palmer said he would love to see a day when citizens ‘gift’ organisations like his access to some of their personal information for the purpose of improving public policy – “kind of like ticking an organ donor box”.

The integration of this richer level of data with the ABS’ existing census stores could have implications for decision making in Australia that are “potentially pretty amazing”, Palmer said.

“We could radically improve our understanding of the state of the Australian economy and society, right down to the individual and household level.

“We could produce more dynamic pictures of population counts such as making distinctions between populations at daytime and night-time which would be very useful during natural disasters," he said.

“With a more detailed economic picture we could forecast the impact of the closure of a factory in a specific town. We would know what particular skill set or employment relationship would be affected, and whether former employees are supported by the people around them or are desperately looking for work.

“It would dramatically transform the way that we make big decisions."

The wealth of customer data currently held by private enterprise could also be used by the ABS, but Palmer said he would be in no hurry to compel them to hand it over, even though Australian legislation enables it.

He would prefer they view it as an exercise in good corporate citizenship.

“If we are going to use this legislation for the ends of informing decision making then we are going to have to have a compelling business case around unlocking public value without trampling over private interests,” he said.