Revealed: IBM error caused NAB outage

A 24-hour outage to National Australia Bank’s banking systems last week was the result of an outsourced IBM staffer deleting a production domain name system (DNS) zone, iTnews can reveal.

At 4:30pm on October 4, NAB’s mobile banking, its online UBank subsidiary, contact centre, some payments processing and corporate systems went offline.

The services remained down for the majority of customers throughout the following day as the bank rushed to restore its systems.

It is yet to provide an explanation for the outage.

However, iTnews can reveal that the downtime was triggered by human error within the outsourced workforce provided by its IT systems and infrastructure partner IBM.

NAB outsourced responsibility for its mainframes, storage and hosting, desktop fleet management, printing, service desks and operating systems to IBM in 2010, transferring 450 positions to the provider.

Despite the bank’s rule that no production changes be made during business hours, an IBM staffer - understood to be located offshore - attempted a change and instead deleted a primary DNS zone.

A primary zone contains the original data for all domains in the zone, and is the only type that can be edited or updated. Changes to a primary zone are replicated to any servers that host secondary or stub versions of the zone.

The problem was identified fairly quickly, sources told iTnews. IBM was able to stop the changes replicating to every server in the DNS, and pulled in fresh copies of the zone to affected servers from those that hadn't yet replicated the changes.

Delays in the services coming back online stemmed from the time-consuming wait for the changes to propagate through the wider DNS, the sources said.

The repercussions for the error are unknown.

NAB declined to comment when approached by iTnews. IBM did not respond to a request for comment by the time of publication.