Telstra reveals Clayton data centre 'master plan'

Lifts lid on retrofits, new builds, key learnings.

Telstra has lifted the lid on the inner workings of data centre modernisation and expansion works at its Clayton, Victoria site, shining a light on the already 4.5 year-plus project.

The carrier's managed data centres general manager Jon Curry told the Australian Data Centre Strategy Summit that the telco took a "master plan approach to the site", which consisted of multiple buildings, several ageing or ripe for updating.

"We sat down with some design engineers and looked at each of the buildings, looked at their age, their purpose, and then decided on how we were going to attack the site from a data centre retrofit and a plan and a build [perspective]," Curry said.

"We wanted to consolidate the number of buildings and the IT load within them."

Curry boiled Telstra's modernisation strategy for the site down to "retrofit where it makes sense, and ... build new where there's space."

Sandpits and heart surgery

Two projects under the broader modernisation works targeted separate floors of an operations building that dated back to 1985.

The first project, codenamed 'the sandpit', involved turning the lower level of the building into a 1200 square metre data centre.

"When they built the building, it was two-storeys, [and] they decided to fit out one floor only, leaving a significant footprint underneath for future growth," Curry said.

"The space I inherited below the operations building ... was about 1700 square metres of sand that they'd filled in after they'd finished the building."

On the floor above the 'sandpit' sat a "live computer room, servicing all of the mobile activation systems, many of the general ledger systems, and many corporate support systems for the company including a small cloud footprint".

"So any issue with construction in this area was going to have a big impact on the organisation, particularly if I dropped load upstairs," Curry said.

Transformation of the lower level from sandpit to data centre took about 18 months. The 1200 square metres of technical space is subdivided into three 400 square metre "zones", with their own independent cooling and power.

"I can take [any] one of those zones offline, do a refurbishment, do some maintenance," Curry said.

Curry noted that the structure that housed the 'sandpit' space imposed some limitations on what could be achieved in the space.

"I was sort of boxed in on how I did the design because of the building fabric and the existing building services," he said.

"We weren't able to implement free air cooling due mostly to the fact that the existing building fabric was tilt slab and we couldn't put any external extrusions in.

"We're also limited to static UPSs [uninterruptible power supplies] because we couldn't get DRUPS [diesel rotary UPS] into the building."

The second project within the operations building targeted the data centre room located above the 'sandpit'.

"I was tasked with upgrading that in situ," Curry said. The project was codenamed 'heart surgery at 36,000 ft'.

Works included replacing 46 direct expansion (DX) computer room air conditioning (CRAC) units with "14 chilled water units, two [past end-of-life] UPS sets with three UPS sets in an N+1 [configuration] and four cooling towers".

Telstra achieved "significant power savings" simply by replacing the CRACs and putting in new chillers and cooling towers.

"I've also been able to increase my IT load by 57 percent and only increase my overall power load by 5 percent, so I've shifted the energy from the mechanical and electrical into IT, significantly improving my PUE [power usage effectiveness]."

Energy overhaul

Further works began in June 2011 to create a new "energy centre" for the site, effectively housing power feeds and infrastructure for the facilities.

"We have had to take the services from the road, ... a standby and primary [high voltage] electrical service, and reticulate them into this building while maintaining the current site load," Curry said.

"It's been the most challenging project I've ever ever run due mostly to the complexity of the SCADA systems and the implementation within a live site.

"Stakeholder management with this project has been quite important."

Curry said that while technologies used in the centre weren't "truly groundbreaking" — again owing to existing building constraints — the equipment was "highly, highly redundant" to cater to the criticality of Telstra's workloads running out of Clayton.

The centre has also been designed to allow "for future energy sources".

"We've built into the design the ability to take a tri-gen plant," Curry noted. The telco has been mulling tri-gen for the site since at least June 2010.

Cloud housing

Work is continuing on a new facility on land that was previously a carpark at the Clayton site.

The facility accounts for about one-eighth of the five-year, $800 million investment that Telstra is making into its cloud infrastructure, a burgeoning growth area for the business.

The new facility, scheduled for completion in 2014, consists of two 1000 square metre data halls in a single building, with power availability at an average 3 kilowatts per square metre.

It will be "full free air cooled", utilise DRUPS and "containment technologies to make sure we hit our PUE targets."

"At the moment we haven't settled on containment technologies," Curry said.

"The building is still 18 months away and I don't want to lock myself into anything now that might preclude me from getting access to a newer technology in 18 months' time."

Curry said interest in the project from stakeholders was high.

"I've probably got 40 or 50 senior executives who are begging to understand delivery timelines, quality, when they can get customers through," he said.

Read on for Telstra's key learnings so far.