SAN failure fast-tracks ATO's cloud shift

The storage area network failure that crippled the Australian Taxation Office last December forced the agency to fast-track its migration to a cloud environment, chief digital officer John Dardo has revealed.

Problems with the ATO’s HPE 3PAR SAN began last December when it suffered a three-day outage to its online services.

A PwC review of the incident later found that a confluence of events had caused the failure, including software bugs on the SAN disk drives and poor fitting of the fibre optic cable feeding the SAN.

The collapse of the SAN network also caused a second five-day outage in February as a result of efforts to replace SAN cabling.

Speaking at the recent Digital and Cloud Government Summit in Sydney, Dardo said that losing both its primary and secondary storage array networks was "the equivalent of losing 3000 virtual servers worth of applications, times two", and had resulted in two months of lost development time.

"We had to compress all the preparation for this tax time into effectively less than a four month period," he said.

He said there were still some applications "running in a sticky tape version", which could explain why the agency has continued to experience disruptions to its systems as recently as earlier this month.

But the unprecedented outage forced the agency to fast-track the migration of some environments to the cloud to ensure a solid platform could serve its application development and production needs.

"We moved ato.gov.au into the cloud in a few days, we moved ATO Online into the cloud within a couple of months. They were all scheduled to go over a financial year, but we did it in days and months," he said.

"We turned on our dev and testing environments for SBR [single business reporting] services within a few days, and within the next couple of weeks we’ll turn on production for cloud for SBR services.

"So we’ve fast-tracked our shift into other environments to give us the resilience we need."

These services were largely hosted on premise prior to moving to the cloud.

A spokesperson said the shift was part of a "long term plan to move some specific services to a cloud platform" that is supported by Amazon Web Services, Microsoft Azure and Macquarie Government.

The AWS and Microsoft contracts have grown by a combined value of more than $10 million over the last six months as a result of increased cloud use.

The tax office has already pledged to rebuild its internal IT infrastructure capability with a focus on resilience and availability following the outage.