HPE storage crash killed ATO online services

A HPE storage network that crashed on Monday caused the loss of 1 petabyte of data and the downing of the Australian Tax Office’s online services.

Early on Monday morning the ATO advised that its online portals, digital services and website were offline following an unspecified “hardware issue”.

The ATO confirmed to iTnews that the issue was triggered by the catastrophic collapse of a hardware storage solution provided by HPE. It is understood the ATO acquired two new HPE 3PAR SANs late last year.

The collapse of the SAN caused the loss of 1PB of data.

The issue was compounded by the failure of back-up systems which did not kick in immediately following the initial outage: iTnews understands the data corruption issues from the primary SAN were replicated onto the second SAN.

The ATO is currently in the process of rebuilding both and restoring from back-up, it is understood.

The tax office emphasised that no sensitive taxpayer data had been compromised.

Acting ATO CIO Steve Hamilton told iTnews it was the first time this problem had been encountered anywhere in the world.

"We are working with HPE to determine the underlying cause. While these investigations are ongoing, we have had to implement alternative recovery procedures that are taking longer to complete," he said.

iTnews understands that the ATO's Siebel CRM and Sharepoint systems were also made unavailable, leaving many of the agency's staff unable to perform their work.

Hamilton said the agency had made alternative work arrangements for staff impacted by the outage.

The agency said the outage was unrelated to scheduled system maintenance over the past weekend: the ATO took its online services for individuals, its mobile app, and its practitioner lodgment service offline from 10pm December 10 to 7am the following morning.

Update 6:40pm: The ATO's main website is back online after two full days of downtime. 

"You may find you’re unable to access certain tools, calculators and our online services while we continue working to restore all of our services," it advised. 

The tax office is still working to bring its other critical systems - like its tax agent portal - back online.

Update 14/12/16 10am: The ATO said its case management systems are back up and running, and it expects to bring its tax agent portal online "later today". 

Other services will be restored "gradually over the coming days". 

It denied ATO staff had been "sitting idle" while it worked to restore its Siebel CRM system.

"Alternative work arrangements were made over the past few days for any staff member who was unable to perform their normal duties due to the outages," Hamilton said in a statement.

"Many staff were not directly impacted by the outages and were able to work as normal. "

It promised to "fast-track" refunds still owed to taxpayers "where we can".

Update 15/12/16 9am: The tax agent portal and business portal have been restored, the ATO said. 

However, it warned users to be patient as it grappled with increased traffic due to people flooding to the systems.

It said it was still working to restore system and application related data that was affected by the outage.