Inside the Pawsey supercomputing centre

Australian academics will soon begin assembling one of Australia’s largest supercomputers in a built-for-purpose facility in Perth, Western Australia.

The Pawsey Centre.

Linked gallery: Photos: Building the Pawsey Supercomputer Centre

The Pawsey Supercomputer Centre, co-funded by the Australian and West Australian Governments, the CSIRO and four universities (Curtin, Edith Cowan, Murdoch, University of Western Australia), will house a petaflop supercomputer capable of handling data processing for scientific endeavours as data and compute-intensive as the initial stages of the Square Kilometre Array project.

The research community has waited three years for its delivery and will be able to harness its power by mid-July. Its assembly will begin within weeks.

iTnews was fortunate enough to watch the construction of this unique facility as builders prepared to hand it over to the research community (see gallery attached).

By July 2013, the centre will house:

• iVEC’s main Cray Petascale Supercomputer (Magnus) for shared use by Australian researchers, starting out as a 69 Teraflop machine running Sandy Bridge processors.
• A second 200 Teraflop CPU-based Cray supercomputer devoted to the real-time operational requirements of the ASKAP (Australian SKA Pathfinder) and Murchison Radio-Astronomy Observatory (MRO) projects.
• A hierarchical storage management system - initially with twin 20 petabyte Spectra Logic tape libraries, plus a six petabyte disk storage system (a re-badged NetApp array provided by SGI).
• A Data Analysis Engine cluster and a visualisation room.

By June 2014, the facility will be upgraded to feature:

• iVEC’s main Cray Petascale Supercomputer (Magnus) will be upgraded to a full Petaflop (a thousand trillion floating points per second) and will run Ivy Bridge processors and accelerators.

The 2MW, $100 million data centre has been uniquely designed - from both aesthetic and functional perspectives - to suit these applications.

“For most computing projects of this scale, you’d usually choose a system, and then design a building,” noted Paul Nicholls, acting executive director of iVEC. “For this project, system procurement and the construction and design of the building happened in parallel. It was a challenge but led to some unique outcomes.”

Built into the side of the hill in Perth’s Technology Park, the design of the facility is inspired by both WA’s natural resources (red iron) and the shape of a circuit board. Designers of the facility have gone to great lengths to make best use of the surrounding environment.

Solar panels on the side and roof of the facility provide shade from the sun and generate 135 kW of energy, while water harvested from the roof is used within the office space of the facility.

Groundwater cooling

Somewhat more radically - the Pawsey Centre is among the first in Australia to attempt to use groundwater to cool its machines.

Upon our visit to the site, construction crews were drilling holes 140 metres into the earth to access the Mullaloo aquifer, a large and free-flowing body of fresh water. Solar-powered pumps will drive this water up from the aquifer at 22 degrees, rotate it through the facility and past both heat exchangers and through the supercomputer’s array of radiator-style cooling equipment, returning to the earth at a warmer 32 degrees.

Researchers are confident that this warmer water will cool off once in flow underground, and not impact the water being extracted for re-entry into the facility.

The operators of the data centre intend to use groundwater and free air cooling as often as possible to avoid the use of large amounts of water from the mains supply. Free air cooling will be used when Perth’s outside temperatures fall below 18 degrees Celsius in an attempt to maintain an ambient air temperature of 24 degrees Celsius within the data halls.

They were hesitant, however, to estimate the savings these measures might generate due to the experimental nature of the project.

A very raised floor

The other distinctive design feature of the Pawsey Centre is its three metre raised floor. All power, cooling and network servicing is at basement level, in a space large enough for technicians to service without walking onto the data room floor.

The facility was designed to run “lights out” (without technicians on site), but features an operations room and facilities that help demonstrate the results of projects run on the supercomputers for researchers.

iVEC’s visualisation room in the main building, for example, will allow researchers to view 3D models of data crunched by the supercomputers.

“You might, for example, conduct a flight over the earth and run magnetic resonance imaging to search for ore deposits, but until you can get a rotatable 3D model for a mining engineer, they can’t visualise the best areas to mine,” Nicholls said. “This is one way supercomputers can generate improved processes.”

Ready for action

Researchers have waited three years for the Pawsey Centre to be built - running code on two ‘pathfinder’ systems, the ‘Epic’ supercomputer at Murdoch University and ‘Fornax’ supercomputer at the University of Western Australia.

“We wanted to help researchers start to use code on supercomputers, and think of bigger ideas, “Nicholls told iTnews.

The Pawsey Centre will take computing in Australian academia to a new level.

“Epic and Fornax were training wheels for building this,” he said.

For more information on iVEC, see www.ivec.org.