SKA researchers probe data vendors

 

Big Data, multiplied.

Research institutions participating in Australia and New Zealand's bid to host the Square Kilometre Array radio astronomy project have begun collaborating with the likes of IBM and Cisco on projects to develop new methods of processing and storing data from the telescopes.

The moves come in an effort to tame what is expected to be one of the most data-intensive projects to date.

The SKA, a 3000-dish array of telescopes, is expected to yield more than an exabyte of raw data per day by 2024, some 99 percent of which will likely be dumped in favour of usable images and time-series information for use by researchers.

Should Australia and New Zealand win rights in February to host the project over South Africa, data from the project will ultimately be held in the 500 square metre spare capacity at the Pawsey Centre high performance computing facility in Perth.

"Historically what they would do is put it on tape or disk and ship it down the mountain and load it up on some labs," said Dougal Watt, IBM New Zealand chief technology officer.

"None of them have approached the level the SKA proposes; nothing is in that league at the moment and there will probably be nothing in that league for a very long time."

The closest parallel, according to Watt, is Europe's Large Hadron Collider, which is estimated to produce approximately 15 petabytes per year. A global computing grid comprising 140 centres in 35 countries was established in 2008 to handle, process and store the data from the scientific experiments run in the LHC. This currently handles peaks at 10 GB of data processed per second.

Phase one of the SKA, to begin construction in 2016, will deliver about a tenth of the array's full capacity by 2019 and require a 100 petaFLOP machine. While a huge amount of data will be processed and disposed of, the project will require petabytes of storage every year.

Methods of processing the data quickly, storing it logically and transferring it quickly between its ultimate home and research institutions continue to be explored by researchers ahead of a full design of the data system due by 2015.

Significant research and development projects have begun between IBM, Cisco and Direct Data Networks to develop methods of implementing a hierarchical file system that will store the data appropriately for researchers to access in minute detail.

The companies had previously signed memoranda of understanding with the global SKA Organisation to contribute to research and development on the project.

"We believe that the storage companies like DDN and IBM, the heirarchical storage systems they're working on are probably adequate in terms of data volume for the SKA," said Peter Quinn, director of the International Centre for Radio Astronomy Research.

"I think where the challenges are going to lie is in data management - how you stage the data in the right way to process it, how you pre-condition the data, how you store the data."

Researchers at ICRAR have focused on the NASA-funded Heirarchical Data Format 5 (HDF5) as a possible file format for the data.

The format, combined with research involving IBM and DDN, would allow the storage systems to store data based on latency and high or low availability requirements for researchers.

The goal is to "effectively try to enable applications to instruct storage on the best way to put the data in place".

"If you don't [optimise], you can have a very intelligent and structured storage system but the storage system doesn't know anything about what the data is going to be used for," Quinn told iTnews.

Also in tow are research projects with Cisco to optimise networking equipment to better respond to the specific data flow patterns rates expected from the SKA during the first phase of the project.

Automated processing

Concurrently, researchers in New Zealand recently completed a three-month prototype of software that automates classification of star systems and other astronomical objects, a process that previously required manual intervention by scientists.

The three-month prototype saw reference data processed on laptops but would be required to scale up massively to ultimately handle the amount of data from the telescopes.

The software, built on the open source International Virtual Observatory Association Ontology, is expected to provide easier processing and storage of data.

"It has the ability to represent peta-size data sets, it would dramatically reduce the amount of data sets you'd need to have stored," IBM's Watt said.

"You'd still need to have the original peta-size data file because at some point you do need to go back and you might need to read about things inside the original image but we think we can get a massive reduction down to tera-sizes."

Though the prototype was considered a success, there have been few further talks on whether the software will be integrated into the wider Australian/New Zealand project.

"There's no one way that this problem of managing that data is going to be solved, it requires I think multiple approaches to try and address the problem," Watt said.

Quinn said ICRAR would continue to pursue further research and development projects over coming years ahead of finalising the systems ultimately required for the project.

Copyright © iTnews.com.au . All rights reserved.


SKA researchers probe data vendors
 
 
 
Top Stories
Frugality as a service: the Amazon story
Behind the scenes, Amazon Web Services is one lean machine.
 
Negotiating with the cloud email megavendors
[Blog post] Lessons from Woolworths’ mammoth migration.
 
Qld govt to move up to 149k staff onto Office 365
Australia's largest deployment, outside of the universities.
 
 
Sign up to receive iTnews email bulletins
   FOLLOW US...

Latest VideosSee all videos »

The great data centre opportunity on Australia's doorstep
The great data centre opportunity on Australia's doorstep
Scott Noteboom, CEO of LitBit speaking at The Australian Data Centre Strategy Summit 2014 in the Gold Coast, Queensland, Australia. http://bit.ly/1qpxVfV Scott Noteboom is a data centre engineer who led builds for Apple and Yahoo in the earliest days of the cloud, and who now eyes Asia as the next big opportunity. Read more: http://www.itnews.com.au/News/372482,how-do-we-serve-three-billion-new-internet-users.aspx#ixzz2yNLmMG5C
Interview: Karl Maftoum, CIO, ACMA
Interview: Karl Maftoum, CIO, ACMA
To COTS or not to COTS? iTnews asks Karl Maftoum, CIO of the ACMA, at the CIO Strategy Summit.
Susan Sly: What is the Role of the CIO?
Susan Sly: What is the Role of the CIO?
AEMO chief information officer Susan Sly calls for more collaboration among Australia's technology leaders at the CIO Strategy Summit.
Meet the 2014 Finance CIO of the Year
Meet the 2014 Finance CIO of the Year
Credit Union Australia's David Gee awarded Finance CIO of the Year at the iTnews Benchmark Awards.
Meet the 2014 Retail CIO of the Year
Meet the 2014 Retail CIO of the Year
Damon Rees named Retail CIO of the Year at the iTnews Benchmark Awards for his work at Woolworths.
Robyn Elliott named the 2014 Utilities CIO of the Year
Robyn Elliott named the 2014 Utilities CIO of the Year
Acting Foxtel CIO David Marks accepts an iTnews Benchmark Award on behalf of Robyn Elliott.
Meet the 2014 Industrial CIO of the Year
Meet the 2014 Industrial CIO of the Year
Sanjay Mehta named Industrial CIO of the Year at the iTnews Benchmark Awards for his work at ConocoPhillips.
Meet the 2014 Healthcare CIO of the Year
Meet the 2014 Healthcare CIO of the Year
Greg Wells named Healthcare CIO of the Year at the iTnews Benchmark Awards for his work at NSW Health.
Meet the 2014 Education CIO of the Year
Meet the 2014 Education CIO of the Year
William Confalonieri named Healthcare CIO of the Year at the iTnews Benchmark Awards for his work at Deakin University.
Meet the 2014 Government CIO of the Year
Meet the 2014 Government CIO of the Year
David Johnson named Government CIO of the Year at the iTnews Benchmark Awards for his work at the Queensland Police Service.
Q and A: Coalition Broadband Policy
Q and A: Coalition Broadband Policy
Malcolm Turnbull and Tony Abbott discuss the Coalition's broadband policy with the press.
AFP scalps hacker 'leader' inside Australia's IT ranks.
AFP scalps hacker 'leader' inside Australia's IT ranks.
The Australian Federal Police have arrested a Sydney-based IT security professional for hacking a government website.
NBN Petition Delivered To Turnbull's Office
NBN Petition Delivered To Turnbull's Office
UTS CIO: IT teams of the future
UTS CIO: IT teams of the future
UTS CIO Chrissy Burns talks data.
New UTS Building: the IT within
New UTS Building: the IT within
The IT behind tomorrow's universities.
iTnews' NBN Panel
iTnews' NBN Panel
Is your enterprise NBN-ready?
Introducing iTnews Labs
Introducing iTnews Labs
See a timelapse of the iTnews labs being unboxed, set up and switched on! iTnews will produce independent testing of the latest enterprise software to hit the market after installing a purpose-built test lab in Sydney. Watch the installation of two DL380p servers, two HP StoreVirtual 4330 storage arrays and two HP ProCurve 2920 switches.
The True Cost of BYOD
The True Cost of BYOD
iTnews' Brett Winterford gives attendees of the first 'Touch Tomorrow' event in Brisbane a brief look at his research into enterprise mobility. What are the use cases and how can they be quantified? What price should you expect to pay for securing mobile access to corporate applications? What's coming around the corner?
Ghost clouds
Ghost clouds
ACMA chair Chris Chapman says there is uncertainty over whether certain classes of cloud service providers are caught by regulations.
Was the Snowden leak inevitable?
Was the Snowden leak inevitable?
Privacy experts David Vaile (UNSW Cyberspace Law and Policy Centre) and Craig Scroggie (CEO, NextDC) claim they were not surprised by the Snowden leaks about the NSA's PRISM program.
Latest Comments
Polls
Which bank is most likely to suffer an RBS-style meltdown?





   |   View results
ANZ
  21%
 
Bankwest
  9%
 
CommBank
  11%
 
National Australia Bank
  17%
 
Suncorp
  24%
 
Westpac
  19%
TOTAL VOTES: 1452

Vote