Corrupted file crashes NAB payments

 

Partner banks Citibank, HSBC also affected.

Errors have been introduced into more than 19,000 National Australia Bank accounts as the bank worked to rectify a "corrupted file" that crashed its payment processing system on Wednesday night.

The four-hour outage delayed "all types of payments" - including Centrelink welfare, credit card, salary and interbank transactions - due into accounts on Thursday November 25.

Transactions made on Friday were also delayed as NAB worked through a backlog of potentially "millions" of accounts.

The bank today stated that it had fixed the glitch, and promised to "put things right" for any customers who had incurred costs from the delays.

However, "duplicate or missing transactions" were introduced in around 19,000 accounts "because the system was under some duress," a spokesman told iTnews this morning.

"Over the course of the weekend, we've managed to rectify the situation," the spokesman said, adding that NAB would rectify any remaining errors today, and overnight if necessary.

Besides delaying payments to and from NAB customer accounts, Wednesday's outage also affected customers of NAB's partner banks, HSBC and Citibank.

NAB processes direct entry and over-the-counter deposits for Citibank, as well as interbank transfers, payroll payments, direct debits, cheque payments and deposits for HSBC.

A Citibank spokesman told iTnews on Friday that it had been in contact with its corporate customers and was responding to retail customer queries about any processing delays.

Citibank anticipated that NAB would quickly rectify the technical issues that resulted from its "infrastructure failure".

HSBC apologised to customers for any inconvenience, urging those who needed immediate access to funds to visit an HSBC branch.

"We're working with NAB to resolve the issue as a matter of urgency and we hope to expedite customer payments as soon as possible," a HSBC spokesman said.

NAB said the outage was a one-off incident, but University of Sydney computer science professor Alan Fekete said sensible enterprise system design would have kept systems operational despite data corruption.

Data could have been stored in multiple places, to assist in recovery and keep systems running should one machine crash or one copy of data be corrupted, he said.

"It's possible to remain viable once a computer glitch kicks in by using a variety of hardware and by keeping records so you can untangle problems after the event," he stated.

"Proper data management stays operational and ensures an audit trail. In this case the system stopped operating when a file became corrupted, suggesting an oversight in the design of the wider IT systems."

Updated at 2.22pm to include Alan Fekete's comments.

Copyright © iTnews.com.au . All rights reserved.


Corrupted file crashes NAB payments
"The fact that IT systems are vulnerable to myriad meltdowns is a reality that anyone with a shred of tech awareness is going to acknowledge. The fact that those systems can impact on so many ..."
By btone
 
 
 
Comments: 8
CyrusLesser
Nov 29, 2010 3:27 PM
Has NAB never heard of rollback and recover? What an incompetent way for a bank to operate. And is APRA asleep at the wheel too?
IanL-S
Nov 29, 2010 3:35 PM
The double processing error was a real pain. I had two transactions double-processed, however this did not happen until the Saturday following the glitch. The double procession was fixed by this morning, however in the process NAB misplaced $20,000 of my money. It appears they reversed a transaction that had not be double-processed. Very poor quality control seems to have been present when trying to correct things.
toner
Nov 29, 2010 4:08 PM
I think we can stop calling this a 'glitch'. A corrupted file crashes the system and you're not able to roll back, restore or anything? Just call it what it is - a stuff up due to sub-standard software (and/or staff) and poor quality control procedures.
alectts
Nov 29, 2010 5:53 PM
This situation is a product of the times. The Queensland Health payroll fiasco, the Treasury Department/Corptech farce, The Brisbane City Council Payroll disaster. The problem is that the REAL experience has left the industry and in their place are the young bloods who are always so certain that 'it won't happen'. Well, from an old hand now displaced from the industry because he's too old (!) whenever you think that it won't - it will! So, the message is - be prepared not only for the unexpected but for the impossible too! I forecast an increase in the number of major IT operational catastrophies until the youngsters grow up - and that could take some time judging by current indicators.
deonast
Nov 29, 2010 8:26 PM
Tough customers here. I didn't mind just monitored my account to make sure it was sorted in the end. @CyrusLesser they did rollback and recover but they aren't a little online store they process a shitload of transactions a roll back and recover means you leave a whole lot of new transactions waiting to be processed. There is some serious computing power having to pull that load.

As for University of Sydney computer science professor Alan Fekete has he been involved in the design of a banking computer system. It seems we have a lot of arm chair critics ready to bash NAB. Really no system can be 100% fool proof as they are all designed by humans.

I'm impressed we have so few incidents like these that we can all jump up and down about it as if it is the end of the world when it happens.

I do think some people have really blown this out of the water for a bit of attention. Take choice magazine, they couldn't help but slam the NAB, and not about product competitiveness like they are supposed to but because it would get them some media exposure for their publication.

No I don't work for NAB, I just have a bit of money with them and I'm glad I wasn't working in IT support for them pity the poor support staff.
Hmmmm
Nov 29, 2010 10:30 PM
Wow..as Deonast says...tough crowd. However, if we examine the fact from hysteria:

On one hand 19,000 affected accounts becomes "millions" - I seem to remember a couple of loaves and a fish doing the same thing around a while back.....

Secondly, poor Professor Fekete (is this an Irish phonetic pronunciation by the way), quoted I am sure out of context - he is quite right, but like saying England's batsmen gave us a lesson today...obvious but not really relevant here.

Thirdly, it is very very annoying and maybe a bit embarrassing (for those refused transactions) for customers and other banks for several days, but in the global context, with Ireland melting down, the European Banking System proposing throwing more Zeros than a Japanese Kamikaze squadron at the European Governments debt situation, and N and S Korea standing on the edge of a precipice - is it that bad?

Come on people - lets remember we are in Aus and get over ourselves here...

PS Alectts - didnt your generation build these systems that are being replaced? Cos they cant handle the new modern reality of business? Just asking....

No - I dont work for NAB either...
Ace
Nov 30, 2010 1:01 AM
Wouldn't we all love to know who this 'corrupt file' is. I'm sure he/she is undergoing a little counselling right now.
btone
Nov 30, 2010 9:23 AM
The fact that IT systems are vulnerable to myriad meltdowns is a reality that anyone with a shred of tech awareness is going to acknowledge. The fact that those systems can impact on so many individuals, many of them old, frail or with no tech awareness aside from inserting a card and pressing a button is the issue. The fact that the NAB posts obscene profits and is answerable to its only master, the 'shareholder', and yet cannot afford to invest in effective backup and contingency systems is a disgrace. The fact that some seem to be relaxed with this farce is a sad reflection on the way we have become programmed to accept tech failure as an acceptable fact of life. Hopefully our nuclear facility in the metropolitan area of Sydney is less prone to such 'acceptable' margins of file corruption error.

Comments have been disabled for this article.
 
 
Top Stories
Photos: HTC One vs Samsung Galaxy S4
Android giants battle it out.
 
Project management lessons from the QLD Health payroll inquiry
Analysis: How not to run a major IT project.
 
Review: Asus Fonepad
Calling on the Big Phone.
 
 
Sign up to receive iTnews email bulletins
   FOLLOW US...

Latest VideosSee all videos »

Bankwest builds continuous delivery capability
Bankwest builds continuous delivery capability
To automatically deploy test/dev sandboxes by mid-year.
Veterans' Affairs sets sights on modernisation
Veterans' Affairs sets sights on modernisation
Data safe with Human Services, CIO says.
Citi Australia drops platform customisations
Citi Australia drops platform customisations
Technology chief shifts focus from building to leveraging systems.
VicRoads restructures IT team
VicRoads restructures IT team
Department moves to align with industry benchmarks.
Zurich Australia extends IT team offshore
Zurich Australia extends IT team offshore
Malaysian staff served from Australian data centres.
Leigh Berrell - Utilities CIO of the Year
Leigh Berrell - Utilities CIO of the Year
Yarra Valley Water CIO Leigh Berrell accepts his Benchmark Award for Utilities CIO of the Year.
Wayne McMahon - Retail CIO of the Year
Wayne McMahon - Retail CIO of the Year
Domino's Pizza CIO Wayne McMahon accepts his Benchmark Award for Retail CIO of the Year.
Inside Perpetual's ongoing IT transformation
Inside Perpetual's ongoing IT transformation
CIO Jenny Levy discusses how outsourcing will help the firm "simplify, refocus and grow".
Managing Complexity - Defence's Daniel McCabe
Managing Complexity - Defence's Daniel McCabe
Daniel McCabe, Assistant Secretary of Australia's Department of Defence, provides the audience at the iTnews Data Centre Strategy Summit with a deep dive into the organisation's data centre consolidation program.
How Facebook designed the data centre from scratch - Marco Magarelli
How Facebook designed the data centre from scratch - Marco Magarelli
The full keynote by Facebook data centre architect Marco Magarelli at the Australian Data Centre Strategy Summit. Magarelli details the design considerations behind the social network's Prineville, Oregon; North Carolina and Luleå, Sweden data centres.
Modernising Legacy Data Centres - Telstra's Jon Curry
Modernising Legacy Data Centres - Telstra's Jon Curry
Telstra general manager of managed data centres Jon Curry guides the audience at the iTnews Australian Data Centre Summit through the build of the telco's Clayton, Victoria data centre.
NSW Government launches NABERS data centre rating tools
NSW Government launches NABERS data centre rating tools
Matthew Clark from the NSW Department of Environment guides facilties managers through the details of the new NABERS data centre energy rating tool at the Australian Data Centre Strategy Summit.
NABERS launch panel: Australian Data Centre Strategy Summit
NABERS launch panel: Australian Data Centre Strategy Summit
Matthew Clark (NSW Dept of Environment), Greg Boorer (Canberra Data Centres), Glenn Allan (National Australia Bank), Mike Andrea (Strategic Directions) and Bob Sharon (Green Global Consulting) discuss the impact of the NABERS data centre rating.
Judges notes: Fortescue Metals [The Benchmark Awards]
Judges notes: Fortescue Metals [The Benchmark Awards]
iTnews' panel of judges discuss Fortescue Metals 'New World of Work" project, one of three shortlisted finalists for the Industrials category of the CIO Benchmark Awards.
Judges notes: Retail [The Benchmark Awards]
Judges notes: Retail [The Benchmark Awards]
iTnews' panel of judges discuss the shortlisted finalists for the Retail category of the CIO Benchmark Awards.
Judges notes: Pacific Aluminium [The Benchmark Awards]
Judges notes: Pacific Aluminium [The Benchmark Awards]
iTnews' panel of judges discuss Pacific Aluminium's lightning fast service desk refresh, one of three shortlisted finalists for the Industrials category of the CIO Benchmark Awards.
Judges notes: Domino's Pizza [The Benchmark Awards]
Judges notes: Domino's Pizza [The Benchmark Awards]
iTnews' panel of judges discuss Domino's Pizza's shift to hosted services, one of three shortlisted finalists for the Retail category of the CIO Benchmark Awards.
Judges notes: McDonald's Australia [The Benchmark Awards]
Judges notes: McDonald's Australia [The Benchmark Awards]
iTnews' panel of judges discuss McDonald's Australia's new self-service portal for employees, one of three shortlisted finalists for the Retail category of the CIO Benchmark Awards.
Judges notes: ING Direct [The Benchmark Awards]
Judges notes: ING Direct [The Benchmark Awards]
iTnews' panel of judges discuss ING Direct's 'Bank in a Box', one of three shortlisted finalists for the banking and finance category of the CIO Benchmark Awards.
Judges notes: Yarra Valley Water [The Benchmark Awards]
Judges notes: Yarra Valley Water [The Benchmark Awards]
iTnews' panel of judges discuss Yarra Valley Water's insourcing project, one of three shortlisted finalists for the Utilities category of the CIO Benchmark Awards.
Latest Comments
Polls
Do you prefer the Coalition's NBN policy?

   |   View results
Yes
  19%
 
No
  81%
TOTAL VOTES: 1643

Vote