NEWS
GOVERNMENT
SECURITY
AI
DIGITAL NATION
RESOURCES
PODCAST
EVENTS
ADVERTISE

BUSINESS CLOUD DATA CENTRE

EDUCATION FINANCIAL SERVICES HARDWARE

HEALTHCARE INDUSTRIAL NETWORKING

PROJECTS SOFTWARE STORAGE

STRATEGY TECHNOLOGY TELCO/ISP

HR MARKETING FINANCE LEADERSHIP GOVERNANCE EMERGING TECH

STATE OF SECURITY 2025 State of HR Tech State of AI 2024

Special Reports Focal Points MEDIA HUB

PARTNER CONTENT PARTNER HUBS RESEARCH

SECURITY AWARDS DIGITAL NATION AWARDS BENCHMARK AWARDS Digital As Usual

NEWS

BUSINESS CLOUD DATA CENTRE EDUCATION FINANCIAL SERVICES HARDWARE HEALTHCARE INDUSTRIAL NETWORKING PROJECTS SOFTWARE STORAGE STRATEGY TECHNOLOGY TELCO/ISP

GOVERNMENT SECURITY AI DIGITAL NATION

HR MARKETING FINANCE LEADERSHIP GOVERNANCE EMERGING TECH

RESOURCES

STATE OF SECURITY 2025 State of HR Tech State of AI 2024 Special Reports Focal Points MEDIA HUB PARTNER CONTENT PARTNER HUBS RESEARCH

PODCAST EVENTS

SECURITY AWARDS DIGITAL NATION AWARDS BENCHMARK AWARDS Digital As Usual

ADVERTISE

Log In

Email:

Password:

Remember me | Forgot password?

Don't have an account? Register now!

Home
Galleries
Technology
Networking

Photos: Building the Pawsey Supercomputer Centre

Australia's research community gets a purpose-built data centre.

on Apr 8 2013 1:57PM

Welcome to The Pawsey Supercomputer Centre in Perth, which will soon house a petaflop supercomputer capable of handling data processing for the Square Kilometre Array amongst other scientific endeavours.

iTnews visited the Pawsey Supercomputer Centre just days before construction crews were scheduled to hand over the keys to the site’s new owners.

The Pawsey supercomputing project is a collaboration between the Australian Government, which stumped up $80 million in capital for the infrastructure, the West Australian Government, which contributed $20 million to support its operations, and shareholder research institutions the CSIRO, Curtin University, Edith Cowan University, Murdoch University and the University of Western Australia.

The $100 million, two megawatt Pawsey Centre consists of two large multi-level buildings, cut into a hill in Perth’s Technology Park precinct. The design of the building uses the reddish iron commonplace in West Australian historical buildings, symbolic of the state’s rich resource base, plus LED-fitted rails that conjure up the idea of a semiconductor. Combined, these illustrate what this supercomputer centre will contribute to the Australian economy - compute power of a scale that can deliver outcomes for both research and industry. From left to right you can see the top floor lobby and office area, which steps down onto three data halls and a separate building at the bottom of the hill containing most of the plant.

On an adjacent block, construction crews work on a radical new idea for reducing the power consumption at the data centre. These crews are drilling 130-140 metres underground to access the Mullaloo aquifer, from which the data centre will extract cool groundwater which will be used to moderate the temperature of a separate water-line that is cooling the supercomputers. This Ground Water Cooling system is a relatively new idea and has the potential to save large quantities of energy and water usage at Pawsey.

The other energy-saving innovation at Pawsey is the use of solar power to generate 135 kW of power. The panels on the roof of the facility should be able to power the pumps for the Ground Water Cooling system. On one side of the building (pictured), this sun shade also captures the sun’s energy - blocking 70 percent of the light and generating around 9 kW. The roof of the supercomputer centre also catches rainwater for use in the toilets.

This is the ‘public area’ of the facility at the top of the site, where researchers can sign-in to enter. Before the build of the Pawsey Supercomputer Centre, researchers had access to two ‘pathfinder’ supercomputers - the 87 teraflop ‘Epic’ system at Murdoch University and the ‘Fornax’ system at the University of Western Australia. The two pathfinders were described as “training wheels” to learn from for the building of the supercomputing centre, which has taken three years from funding to design and construction.

From the public area, researchers can visit the iVEC visualisation room. This will help researchers and other interested parties grapple with the complex calculations computed in the belly of the building. This room will soon feature a large, rear projection screen for visualisation of large data sets, such as those generated by researchers from the National Geo-Sequestration Laboratory, scientists working on advanced computational chemistry projects, or analyses of the universe performed by Australia’s radio astronomers. Data will be fed into the Data Analysis Engines within the supercomputer, and the results presented on-screen.

A view from the top floor of the building (which houses the foyer, office space, an operations room and staging room), down onto the largest of three data halls. The Pawsey Centre boasts 1000 square metres of white space in total, which sits on a very deep (three metre) raised floor, with most cooling and power infrastructure (including PDUs, or Power Distribution Units) housed underneath.

Looking at the same main data hall, from down on a service corridor. By July 2013, this will house iVEC’s main Cray Petascale supercomputer (PS1). In Phase One, it will be a 69 Teraflop machine running Sandy Bridge processors, but by Phase Two (June 2014) will be expanded to a full Petaflop and run Ivy Bridge processors and accelerators. In the same room, Pawsey will host a second CPU-based Cray supercomputer devoted to the real-time operational requirements of the ASKAP (Australian SKA Pathfinder) and Murchison Radio Observatory radio astronomy projects (200 Teraflops). Storage and switching for the supercomputer is housed in adjacent rooms (see next slide).

This is the adjacent I/O cell - which connects the supercomputers within the facility to Australia’s Academic Research Network (AARNET) via Cisco Nexus Switches and Palo Alto Firewalls. This I/O cell and the third data hall (not in shot) share Pawsey’s hierarchical storage management system - initially twin 20 petabyte SpectraLogic tape libraries (which will potentially reach a 100 petabyte capacity), plus a six petabyte disk storage system (a re-badged NetApp array provided by SGI).

The operations room will feature several large screens and four workstations, but to all intents and purposes the facility has been designed to run without humans needing to be on-site. Considering the level of interest and anticipation from the research community, designers of the facility remark that it is more likely to be run as a “dim” facility than fully “lights out”.

The technical set-up area connects the loading dock on one side of the building, an adjacent service aisle for the data centre floor and the operations room via a set of stairs.

Welcome to the belly of the beast. We’re now at basement level, underneath the data halls. Here you can see PDUs feeding power up to the supercomputer and storage kit above.

Note that the rails for connectivity and power are underneath the data hall floor, not above. The designers of the facility felt it would be easier to service and upgrade power and water in this three metre cavity. The facility will feature very little copper connectivity, relying instead on optical fibre connectivity.

This complex series of pipes connect to air handling units underneath the data hall. The supercomputers are water-cooled. A line of 16 degree Celsius water enters the data hall and into a series of radiators affixed to the racks. Two big fans at one end of each rack blow air through sequences of servers and radiators such that the air coming out at the other end is as cool or cooler than the ambient temperature in the room, which the operators aim to keep at around 24 degrees Celsius. Some 90 percent of the heat generated by the machines will be extracted via this water-cooling method. The air handling units also feature an economy cycle to switch off and extract outside air when Perth’s temperature falls below 18 or 19 degrees. This should provide 10-15 percent of the facility’s cooling requirements.

The air handling units and power distribution units under the data halls are fed via a tunnel by plant in a separate building at the bottom of the property. Out of shot there are several attached rooms where power enters the building, is fed into transformers, a switching room and into RUPS before distribution to the data centre. RUPS are like a series of big flywheels that spin from the current to produce a cleaner current of their own (smoothing out surges or feedback). The inertia of this movement under full load allows the RUPS units to continue to power the data centre for 20 seconds after the mains have been cut. Being that the supercomputing workloads aren’t ‘mission critical’, Pawsey contains no diesel generators. The RUPS - and an attached battery system - simply provide enough time to cool the equipment down and safely switch machines off without the risk of corruption. There is, however, a redundancy requirement around water. Water is sourced from the mains supply and cooling towers (see next slide) into one set of heat exchangers, or from Pawsey’s ground water bores into a separate set of heat exchangers. Either option has the capacity to manage 100 percent of the facility’s needs.

This is a shot of the heat exchangers handling the ground water cooling system. Explaining this system without a diagram is tricky! Basically you have three independent paths for the water, which never actually physically touch but impact each other’s temperature via the use of heat exchangers (like a large radiator). One independent path runs between the supercomputers and a heat exchanger - which in turn connects to one of two other independent paths - one from the mains via cooling towers, and one via the ground water system. The cooler water from these two paths helps to reduce the temperature of the water that has come out of the supercomputer.

Our tour ends at the cooling towers. These will help cool the water until the Ground Water Cooling system is up and running. These cooling towers throw water up in the air, the movement of which cools the water that falls back to the bottom of the tank to be extracted into the pipelines for use in the data centre. Ground Water Cooling will save volumes of water because whenever you throw water into the air, you lose water through evaporation. Cooling towers like these can use up to 250,000 litres per day - a 30 to 40 percent water loss depending on humidity and heat.

Linked Articles

Inside the Pawsey supercomputing centre

Tags:

Cricket Australia to deploy SD-WAN, Teams Calling

Defence trials AI radiocomms deception technology

Lake Macquarie City Council resets IT leadership with CTO role

Partner Content

AI Copilot: Breaking Down Silos & Securing the Future

Partner Content Machine identity a key priority for organisations’ security strategies: CyberArk

Microsoft Copilot Partner Hub

Partner Content Australian organisations must act on security – or risk AI ambitions falling flat

Promoted Content AI in cybersecurity: weapon or shield?

Partner Content AI Supercharged: How Search is Powering the Future

Dave Stevens on Brennan's evolution and the need for Aussie tech unity

Sydney's ITKnocks on contact centre AI and the slow death of the IVR

"It's an exciting time to be part of the health and aged care sector"

Insicon founder Matt Miller on the coming 'tsunami' of compliance and educating boards about cyber security

Orro claims Australia first with managed digital asset discovery service

Latest Comments

Events

EDUtech AU
Security Exhibition & Conference 2025
Integrate Expo 2025
Digital As Usual Cybersecurity Roadshow: Brisbane edition
iTnews Benchmark Security Awards 2025

Most popular tech stories

Coles merges digital and customer leadership

Digital Nation announces finalists for its inaugural Digital Nation Awards

Orica to set new workforce systems live in Australia in July

Westpac hires CBA's data chief to lead AI, data and digital

David Jones shapes store design, lease negotiations with customer feedback
Australian MSP Index launched

Ingram Micro Ushers in the Age of Ultra

Announcing Pipeline 2025 tickets, theme & initial speakers

Vic gov to spend $100m on cyber security

"Waiting until an incident is in progress is too late": MSPs urged to review ransomware response as new reporting rules commence
Blackberry celebrates "giant step forward"

The three steps to industrial IoT success

How IBM aims to protect food supplies using IoT, blockchain and agritech

Photos: IoT Impact 2023 brings together data-enabled productivity, sustainability and trust opportunities

Perth IoT vendor Digital Matter names new chief executive

OTHER TECH SITES: techpartner.news | Digital Nation | IoT Hub

All rights reserved. This material may not be published, broadcast, rewritten or redistributed in any form without prior authorisation.
Your use of this website constitutes acceptance of nextmedia's Privacy Policy and Terms & Conditions.