Corrupted file crashes NAB payments

 

Partner banks Citibank, HSBC also affected.

Errors have been introduced into more than 19,000 National Australia Bank accounts as the bank worked to rectify a "corrupted file" that crashed its payment processing system on Wednesday night.

The four-hour outage delayed "all types of payments" - including Centrelink welfare, credit card, salary and interbank transactions - due into accounts on Thursday November 25.

Transactions made on Friday were also delayed as NAB worked through a backlog of potentially "millions" of accounts.

The bank today stated that it had fixed the glitch, and promised to "put things right" for any customers who had incurred costs from the delays.

However, "duplicate or missing transactions" were introduced in around 19,000 accounts "because the system was under some duress," a spokesman told iTnews this morning.

"Over the course of the weekend, we've managed to rectify the situation," the spokesman said, adding that NAB would rectify any remaining errors today, and overnight if necessary.

Besides delaying payments to and from NAB customer accounts, Wednesday's outage also affected customers of NAB's partner banks, HSBC and Citibank.

NAB processes direct entry and over-the-counter deposits for Citibank, as well as interbank transfers, payroll payments, direct debits, cheque payments and deposits for HSBC.

A Citibank spokesman told iTnews on Friday that it had been in contact with its corporate customers and was responding to retail customer queries about any processing delays.

Citibank anticipated that NAB would quickly rectify the technical issues that resulted from its "infrastructure failure".

HSBC apologised to customers for any inconvenience, urging those who needed immediate access to funds to visit an HSBC branch.

"We're working with NAB to resolve the issue as a matter of urgency and we hope to expedite customer payments as soon as possible," a HSBC spokesman said.

NAB said the outage was a one-off incident, but University of Sydney computer science professor Alan Fekete said sensible enterprise system design would have kept systems operational despite data corruption.

Data could have been stored in multiple places, to assist in recovery and keep systems running should one machine crash or one copy of data be corrupted, he said.

"It's possible to remain viable once a computer glitch kicks in by using a variety of hardware and by keeping records so you can untangle problems after the event," he stated.

"Proper data management stays operational and ensures an audit trail. In this case the system stopped operating when a file became corrupted, suggesting an oversight in the design of the wider IT systems."

Updated at 2.22pm to include Alan Fekete's comments.

Copyright © iTnews.com.au . All rights reserved.


Corrupted file crashes NAB payments
"The fact that IT systems are vulnerable to myriad meltdowns is a reality that anyone with a shred of tech awareness is going to acknowledge. The fact that those systems can impact on so many ..."
By btone
 
 
 
Comments: 8
CyrusLesser
Nov 29, 2010 3:27 PM
Has NAB never heard of rollback and recover? What an incompetent way for a bank to operate. And is APRA asleep at the wheel too?
IanL-S
Nov 29, 2010 3:35 PM
The double processing error was a real pain. I had two transactions double-processed, however this did not happen until the Saturday following the glitch. The double procession was fixed by this morning, however in the process NAB misplaced $20,000 of my money. It appears they reversed a transaction that had not be double-processed. Very poor quality control seems to have been present when trying to correct things.
toner
Nov 29, 2010 4:08 PM
I think we can stop calling this a 'glitch'. A corrupted file crashes the system and you're not able to roll back, restore or anything? Just call it what it is - a stuff up due to sub-standard software (and/or staff) and poor quality control procedures.
alectts
Nov 29, 2010 5:53 PM
This situation is a product of the times. The Queensland Health payroll fiasco, the Treasury Department/Corptech farce, The Brisbane City Council Payroll disaster. The problem is that the REAL experience has left the industry and in their place are the young bloods who are always so certain that 'it won't happen'. Well, from an old hand now displaced from the industry because he's too old (!) whenever you think that it won't - it will! So, the message is - be prepared not only for the unexpected but for the impossible too! I forecast an increase in the number of major IT operational catastrophies until the youngsters grow up - and that could take some time judging by current indicators.
deonast
Nov 29, 2010 8:26 PM
Tough customers here. I didn't mind just monitored my account to make sure it was sorted in the end. @CyrusLesser they did rollback and recover but they aren't a little online store they process a shitload of transactions a roll back and recover means you leave a whole lot of new transactions waiting to be processed. There is some serious computing power having to pull that load.

As for University of Sydney computer science professor Alan Fekete has he been involved in the design of a banking computer system. It seems we have a lot of arm chair critics ready to bash NAB. Really no system can be 100% fool proof as they are all designed by humans.

I'm impressed we have so few incidents like these that we can all jump up and down about it as if it is the end of the world when it happens.

I do think some people have really blown this out of the water for a bit of attention. Take choice magazine, they couldn't help but slam the NAB, and not about product competitiveness like they are supposed to but because it would get them some media exposure for their publication.

No I don't work for NAB, I just have a bit of money with them and I'm glad I wasn't working in IT support for them pity the poor support staff.
Hmmmm
Nov 29, 2010 10:30 PM
Wow..as Deonast says...tough crowd. However, if we examine the fact from hysteria:

On one hand 19,000 affected accounts becomes "millions" - I seem to remember a couple of loaves and a fish doing the same thing around a while back.....

Secondly, poor Professor Fekete (is this an Irish phonetic pronunciation by the way), quoted I am sure out of context - he is quite right, but like saying England's batsmen gave us a lesson today...obvious but not really relevant here.

Thirdly, it is very very annoying and maybe a bit embarrassing (for those refused transactions) for customers and other banks for several days, but in the global context, with Ireland melting down, the European Banking System proposing throwing more Zeros than a Japanese Kamikaze squadron at the European Governments debt situation, and N and S Korea standing on the edge of a precipice - is it that bad?

Come on people - lets remember we are in Aus and get over ourselves here...

PS Alectts - didnt your generation build these systems that are being replaced? Cos they cant handle the new modern reality of business? Just asking....

No - I dont work for NAB either...
Ace
Nov 30, 2010 1:01 AM
Wouldn't we all love to know who this 'corrupt file' is. I'm sure he/she is undergoing a little counselling right now.
btone
Nov 30, 2010 9:23 AM
The fact that IT systems are vulnerable to myriad meltdowns is a reality that anyone with a shred of tech awareness is going to acknowledge. The fact that those systems can impact on so many individuals, many of them old, frail or with no tech awareness aside from inserting a card and pressing a button is the issue. The fact that the NAB posts obscene profits and is answerable to its only master, the 'shareholder', and yet cannot afford to invest in effective backup and contingency systems is a disgrace. The fact that some seem to be relaxed with this farce is a sad reflection on the way we have become programmed to accept tech failure as an acceptable fact of life. Hopefully our nuclear facility in the metropolitan area of Sydney is less prone to such 'acceptable' margins of file corruption error.

Comments have been disabled for this article.
 
 
 
Top Stories
Australian miners send drones to work
In-depth: Unmanned aerial vehicles in the resources sector.
 
The New Zealand telco problem
Opinion: Could Telstra save Kiwi telcos?
 
IT price probe to 'name and shame' gougers
Industry ducking the issue, committee claims.
 
Sign up to receive iTnews email bulletins
   FOLLOW US...

Latest VideosSee all videos »

Latest Comments
Polls
Should the Government enact new legislation to protect copyright holders in the digital age?

   |   View results
Yes
  20%
 
No
  80%
TOTAL VOTES: 507

Vote