More weekend outages as the ATO works to fix digital services

The Australian Taxation Office’s digital services will be unavailable for the fourth weekend in a row starting from Friday night, as the agency struggles to remediate its systems after a devastating two-day crash in December.

Online services will be taken down between 11:15 pm (AEDT) on Friday night and 9:00 am on Sunday morning to allow the ATO’s IT team to conduct “critical maintenance” as they “continue to restore ... systems to full functionality” according to advice on the agency's website.

Portals & online services will be unavailable from 11.15pm AEDT Friday 13 Jan. We expect they’ll be back online on Sunday morning, 15 Jan.

— ato.gov.au (@ato_gov_au) January 11, 2017

We appreciate your understanding as we continue the complex task of fully restoring and stabilising our systems.

— ato.gov.au (@ato_gov_au) January 11, 2017

The planned outage follows similar maintenance work between 6-8 January, 30-31 December and 26-27 December.

After last weekend’s disconnection, the ATO advised via Twitter it had “made significant steps towards stabilising systems on the weekend, but more work is required".

The agency is working to stabilise its systems after a catastrophic hardware failure wiped out its full suite of digital services for two days between 12 and 13 December last year.

The outage was caused by the collapse of an HPE SAN device, which was compounded by the subsequent failure of backup systems when corrupted data was replicated to a secondary SAN.

The saga left the ATO with the considerable task of restoring 1PB of data from backup storage to get online services working again.

However the job appears to be taking longer than anticipated.

The agency suffered a second, three-hour outage on 23 December, and appears to have been working furiously through the weekend and Christmas break ever since to fix whatever errors still remain.

A spokesperson for the agency, however, refused to comment on what work still needs to be done or how long the weekend planned outages might have to continue.

On 6 January the agency tweeted that “the process to return our services to normal is complex & time-consuming. We appreciate your patience & understanding as work continues”.

The reportedly ‘unprecedented’ hardware collapse will be the subject of an independent review to assess what went wrong, how to prevent it happening again, and the effectiveness of response by the ATO's vendor partners, including HPE.