Massive CBA outage traced to failed infrastructure upgrade

A massive 12-hour outage at the Commonwealth Bank of Australia that left millions of Australians unable to directly transfer or receive funds on Thursday has been traced to an errant infrastructure upgrade that hobbled the institution’s real-time transactions platform.

In a conspicuous hardware-related failure of the CBA’s previously lauded core system, the institution on Friday was facing-up the grim prospect that non-CBA customers meant to receive money from Australia’s biggest bank may have the wait until early next week for funds to clear.

But why, of all times, the CBA rolled an upgrade on a Thursday – usually a no-go period for fixes because of the high number of payroll runs – is a very live and open question.

So far the bank is only offering words of contrition for the outage, but a number of factors already in the public domain surrounding its tech stack offer some pointers.

In terms of infrastructure, CBA is known to be a heavyweight Dell user across its platforms. And like most banks, it’s also a hefty Oracle user.

As reported by iTnews, Oracle shipped a very large critical patch update earlier this week, some of which carried vulnerability scores as high as 9.8 and 10 and affected financial services products.

Sleep deprived staff at the CBA declined to talk to the infrastructure or security issues, saying instead that the bank took responsibility for the outage and was working to resolve problems and issues for customers hit by the mammoth glitch.

 “We have been working to restore the network systems overnight and they are now returning to normal,” a CBA spokesperson said.

“Some payments may be delayed appearing in accounts. We are working through the backlog as quickly as possible, but rest assured that all payments both into and out of customer accounts will be properly processed.

“The internal system issues related to an upgrade implemented on Thursday.”

Other banks are hawkishly watching CBA’s problems, with one senior bank source confirming checks to see if the CBA’s problems could re-occur elsewhere.

“Everyone feels this,” the source said.

It is understood that while CBA’s functionality started to restore around 1am on Friday, the bank tested until around 6am before declaring them back up.

An urgent incident autopsy is also underway as the CBA moves to ensure the problems do not repeat.

Notably, eftpos and ATM services were not hit by the outage and continued to function. While that may have previously provided some salve, what quickly became apparent is that CBA’s customers are heavy users of direct transfers, both to each other and via the New Payments Platform.

It is understood the bank moved to disconnect itself from the NPP during the outage, meaning transactions using PayID or overlay services like Osko had connections to CBA taken down.

Ironically, because CBA’s systems are real time, it will be able to process the backlog of payments between its own customers or other institutions capable of fast settlements quickly throughout Friday.

However customers of banks using batched transactions, which are the most common way to move chunky drops like payroll or scheduled bill payments, could have to wait until Monday or Tuesday before funds flicked across by the CBA clear because many do not process over weekends.

Batch processing uses a protocol dubbed T+2 (transaction plus two days) because runs normally cycle through a 48 hours window.

Institutions have been gradually moving to newer protocol called T+0 for some payments, but distribution is still patchy.

Most so-called neobanks are already real-time because they are greenfields builds that have never been near a mainframe. Those institutions will no doubt dine out on the CBA’s misery to try and entice over angry customers. They will also likely snare other refugees from the Big Four.

The biggest headache in terms of payment delays will be payroll, which again raises questions as to why CBA rolled an upgrade on a Thursday.

On Thursday night contractors working for the NSW government received emergency messages from labour hire and recruitment agencies warning them that while payrolls had been lodged, it was not clear when they would clear.

While Westpac is the NSW government’s transactional bank, and employees on the state’s direct payroll are unlikely to have pay delays, pay deposited by external providers using the CBA could be delayed over the weekend because it will have missed batch runs at receiving banks.

CBA’s competitors could also choose to run extra batches to more quickly clear funds, however the problem for all banks is that the weekend is the normal system maintenance window because of lower volumes.

Banks are required to formally report outages – many of which occur without media coverage – to the Reserve Bank of Australia, which has started keeping a log of incidents to guard against a lack of resilience becoming a systemic risk.

After CBA’s Thursday incident, those risks will once again be sharply in focus.