CBA adds 1000 racks in data centre modernisation

CBA has installed displacement ventilation for two new data halls at one of its major Sydney facilities, which will enable it to cool spot loads of up to 30kW per rack without special treatment.

Credit: Joshua Lundberg

Linked gallery: Photos: Inside CBA's modernised Sydney data centre

The bank today opened the two data halls, which join existing “legacy floors” of data centre space in a four-storey facility.

The new halls collectively add 2000m2 of data floor and 1000 racks of capacity.

The halls form part of a major modernisation program for the 19,000m2 facility, which CBA began in 2012. The facility is owned by CBA but operated by outsourcer HPE.

The data centre itself is 27 years old, and the modernisation of building systems together with the addition of two new halls is expected to add another 15 to 20 years to its expected lifespan.

One of the key features of the new halls is displacement ventilation, which uses a wall of large but comparatively slow-moving fans to flood the room with cold air, with hot air contained and exhausted via the ceiling.

Each data floor is expected to house a mix of rack densities from the standard 2-3kW up to dense 30kW racks, but CBA indicated it would not need to contain or spot cool high-density zones.

The bank used computational fluid dynamics (CFD) modelling to test the effect of displacement ventilation on dense racks, and concluded that it could handle spot loads.

For now, CBA is keeping the operating temperature of its new halls at 22 degrees Celsius – its existing standard – but is understood to be looking to raise the operating temperature in line with Ashrae operating standards, subject to the approval of internal IT and its outsourcing partners.

Aside from cooling, the data centre has also been given an access security overhaul with iris scanners on all doors, and has LED lighting that is controlled by motion and thermal sensors.

CBA has shifted to a 2N redundancy model for back-up generators (instead of N+1) and has switched to plant that is modular and hot-swappable, making it more resilient in the event of a failure.

“We’ve moved away from a very monolithic, traditional ‘three nines’ design to a contemporary, ultra-modular design,” a spokesperson said.

CBA ran 1.1MW of simulated IT load testing on the new data halls, and tested 110 separate failure scenarios over a fortnight before declaring the new floor space ready for production workloads.

Many of the workloads that are earmarked to run in the newer halls will come from existing “legacy” floors and halls in the same building.

CBA said it plans to “clean out the legacy floors over time”. That will free up additional space should it be required, but the bank is also open to “cold shelling” the space if there’s no requirement for it.

Existing systems, however, wouldn’t be moved until they had reached their end-of-life or were otherwise in line for an uplift.

Production workloads including CBA’s data warehouse, storage and ATM/POS environments presently sit in an older hall but will not be moved until they need replacing.

Some of these systems were installed while the new data halls were being built and the facility being modernised, meaning their migration into the newer halls could be at least several years away.

CBA is hoping that the new cooling system in the new data halls, in particular, will help the facility achieve a power usage effectiveness (PUE) reduction from 1.6-1.7 to 1.3-1.4.

To cloud or not to cloud

CBA's group executive and CIO David Whiteing said that at the time the decision to modernise was made, the bank was examining its options around the cloud.

"At the time we were making the decision, we didn't have a strong sense of where the regulator would be around cloud and what we could put into the cloud," he said.

"I think the regulator is starting to entertain a more coherent conversation around the cloud, but even so I can't see a world in the next five-to-ten years where we still wouldn't need to have our own data centre capability."

Whiteing said that CBA's data storage growth remains "phenomenal", and that alone is one reason he believes data centres will continue to be needed - to house the ongoing demand for storage.

However, he also sees strategic reasons to keep hold of CBA's data centre infrastructure.

"This is the 'factory' where we manufacture product," Whiteing said. 

"This is the heart of the company. I'd be very reluctant to give that up."