IT outsourcer's US$5m storage disaster

IT outsourcer Northrop Grumman has agreed to pay US$5 million in compensation to a US government customer for a massive systems failure caused by an electrical fault.

The outage last August affected 27 agencies in the US state of Virginia, including the Department of Motor Vehicles. 

An audit released this February blamed Northrop Grumman, which now "will pay the full costs incurred by the state during last year's outage,” the state's Governor Bob McDonnell heralded.

While the state had top-of-the-line equipment, such as EMC's DMX-3 SAN and HP's server monitoring tool HP OpenView, Northrop Grumman fell short on systems implementation and back up procedures, according to the audit.

Neither Northrop Grumman nor Virginia’s own investigators have been able to identify the primary cause of the incident besides a failure of its EMC DMX-3 storage area network that was triggered by an electrical fault. 

No one knows exactly when the error occurred either, however the outage itself was triggered by an “erroneous decision” by an Northrop Grumman subcontractor regarding which component to repair first, according to the audit summary.

Northrop was also blamed for failing to take point-in-time copies of data for restoration while the SAN was being repaired, causing a loss of system data. 

For the next two years Northrop Grumman will give Virgina monthly invoice credits totaling US$1.908 million, spend US$2.09 million making operational improvements to enterprise systems, and will implement recommendations by the audit, such as snapshot systems, valued in total at US$750,000. 

It also paid for the US$250,000 audit which spotted its short-comings.