Telstra reveals Clayton data centre 'master plan'

Telstra has lifted the lid on the inner workings of data centre modernisation and expansion works at its Clayton, Victoria site, shining a light on the already 4.5 year-plus project.

The carrier's managed data centres general manager Jon Curry told the Australian Data Centre Strategy Summit that the telco took a "master plan approach to the site", which consisted of multiple buildings, several ageing or ripe for updating.

"We sat down with some design engineers and looked at each of the buildings, looked at their age, their purpose, and then decided on how we were going to attack the site from a data centre retrofit and a plan and a build [perspective]," Curry said.

"We wanted to consolidate the number of buildings and the IT load within them."

Curry boiled Telstra's modernisation strategy for the site down to "retrofit where it makes sense, and ... build new where there's space."

Sandpits and heart surgery

Two projects under the broader modernisation works targeted separate floors of an operations building that dated back to 1985.

The first project, codenamed 'the sandpit', involved turning the lower level of the building into a 1200 square metre data centre.

"When they built the building, it was two-storeys, [and] they decided to fit out one floor only, leaving a significant footprint underneath for future growth," Curry said.

"The space I inherited below the operations building ... was about 1700 square metres of sand that they'd filled in after they'd finished the building."

On the floor above the 'sandpit' sat a "live computer room, servicing all of the mobile activation systems, many of the general ledger systems, and many corporate support systems for the company including a small cloud footprint".

"So any issue with construction in this area was going to have a big impact on the organisation, particularly if I dropped load upstairs," Curry said.

Transformation of the lower level from sandpit to data centre took about 18 months. The 1200 square metres of technical space is subdivided into three 400 square metre "zones", with their own independent cooling and power.

"I can take [any] one of those zones offline, do a refurbishment, do some maintenance," Curry said.

Curry noted that the structure that housed the 'sandpit' space imposed some limitations on what could be achieved in the space.

"I was sort of boxed in on how I did the design because of the building fabric and the existing building services," he said.

"We weren't able to implement free air cooling due mostly to the fact that the existing building fabric was tilt slab and we couldn't put any external extrusions in.

"We're also limited to static UPSs [uninterruptible power supplies] because we couldn't get DRUPS [diesel rotary UPS] into the building."

The second project within the operations building targeted the data centre room located above the 'sandpit'.

"I was tasked with upgrading that in situ," Curry said. The project was codenamed 'heart surgery at 36,000 ft'.

Works included replacing 46 direct expansion (DX) computer room air conditioning (CRAC) units with "14 chilled water units, two [past end-of-life] UPS sets with three UPS sets in an N+1 [configuration] and four cooling towers".

Telstra achieved "significant power savings" simply by replacing the CRACs and putting in new chillers and cooling towers.

"I've also been able to increase my IT load by 57 percent and only increase my overall power load by 5 percent, so I've shifted the energy from the mechanical and electrical into IT, significantly improving my PUE [power usage effectiveness]."

Energy overhaul

Further works began in June 2011 to create a new "energy centre" for the site, effectively housing power feeds and infrastructure for the facilities.

"We have had to take the services from the road, ... a standby and primary [high voltage] electrical service, and reticulate them into this building while maintaining the current site load," Curry said.

"It's been the most challenging project I've ever ever run due mostly to the complexity of the SCADA systems and the implementation within a live site.

"Stakeholder management with this project has been quite important."

Curry said that while technologies used in the centre weren't "truly groundbreaking" — again owing to existing building constraints — the equipment was "highly, highly redundant" to cater to the criticality of Telstra's workloads running out of Clayton.

The centre has also been designed to allow "for future energy sources".

"We've built into the design the ability to take a tri-gen plant," Curry noted. The telco has been mulling tri-gen for the site since at least June 2010.

Cloud housing

Work is continuing on a new facility on land that was previously a carpark at the Clayton site.

The facility accounts for about one-eighth of the five-year, $800 million investment that Telstra is making into its cloud infrastructure, a burgeoning growth area for the business.

The new facility, scheduled for completion in 2014, consists of two 1000 square metre data halls in a single building, with power availability at an average 3 kilowatts per square metre.

It will be "full free air cooled", utilise DRUPS and "containment technologies to make sure we hit our PUE targets."

"At the moment we haven't settled on containment technologies," Curry said.

"The building is still 18 months away and I don't want to lock myself into anything now that might preclude me from getting access to a newer technology in 18 months' time."

Curry said interest in the project from stakeholders was high.

"I've probably got 40 or 50 senior executives who are begging to understand delivery timelines, quality, when they can get customers through," he said.

Read on for Telstra's key learnings so far.

While the technical details of the modernisation are impressive, equally so are the project, stakeholder and process considerations.

Curry noted that 1500 individual changes had so far been documented and approved across the modernisation works.

"A change at Telstra requires significant approval from the executive level all the way through our network business, and usually requires about three or four hours of documentation, so running 1500 individual changes [represents] a significant amount of time," he said.

Running concurrent retrofit and construction projects demanded "incredible attention to detail", as well as prioritisation of what was most important as the site continued to operate on a business-as-usual basis.

"Managing the day-to-day with two and sometimes three construction programs requires incredible attention to detail," Curry said.

"The construction program managers will always tell you that they're more important than each other.

"You really have to sit down and assess the risk to the business, the risk to the site, and then the risk to the program in that order.

"Security and stability are our highest priority and they always take precedence over the program or any other changes that are trying to be pushed through the site."

Curry noted that one key learning from the modernisation had been to get designs right the first time.

"When you're in the throes of design and you've got a lot of pressure to build out significant data centre footprints or refurbish significant areas of your data centre, there is always an urgency to move to the next step," he said.

"My experience is to take a breath, slow down, get the design right, and ensure that when you're moving to implementation that you've done all the hard yards on the design and you're not getting to a point where you're having to redesign a chilled water loop or increase the size of CRAC units or chillers because somebody missed something."

Curry also cited the importance of getting executives to make fast decisions to prevent any delays to overall timelines.

"Focus on the end game, [on] getting to where you need to get and the date you need to get there, and focus on getting your executives to make decisions very quickly," he said.

"It's incredibly expensive, this game. I can certainly see the amount of money going out of my pocket.

"The faster you make decisions, the faster you get implementation, the quicker you stop the cash flowing out of the organisation on things like delays."