DLA Piper paid 15,000 hours of IT overtime after NotPetya attack

Law firm DLA Piper has revealed its IT team put in 15,000 hours of paid overtime to recover from the NotPetya malware infection.

The company was also forced to wipe its entire Windows environment and “start afresh” after the first two weeks showed nothing in the existing environment was “salvageable”.

The destructive NotPetya malware took a number of large companies offline in June 2017, including DLA Piper, Maersk and TNT Express.

Maersk and TNT Express pegged their losses at $378 million and $374 million respectively; Maersk had to reinstall its entire IT environment in 10 days to recover.

It is now clear DLA Piper’s infection was also extremely costly, exposing weaknesses in their security posture and crisis plans due to the “unprecedented” impact of the malware infection.

“We were hit through a supplier of ours,” DLA Piper’s Melbourne-based regional IT manager Dylan James said.

“The impact of it was very widespread and the recovery from there became quite complex for us and very, very time consuming.

“The first 48 hours were definitely the hardest. Because it was a global attack, every data centre and Windows-based server that we had was impacted.

“It took us literally 48 hours to find a working copy of a domain controller that we could even use to start the recovery.”

James said that IT in the United Kingdom had identified the attack within about 20 minutes of it starting.

However, he said the company’s “flat network structure globally” allowed the malware to easily spread.

“One of the things we’re in the process of doing right now is segmenting our network, separating off our offices and isolating them so that should we get hit again in the future we’ve got a greater chance of containing the spread of the attack rather than being as open as we were on this occasion,” James said.

With a large-scale outage of its core systems, James said it was all hands on deck for the global IT team.

“The first three weeks of the cyber attack we recorded about 15,000 hours of overtime which the organisation paid,” James said.

“[Even so], you rely on the goodwill [of the IT staff] to front up every day and work 20 or 22 hours or whatever they were doing day after day, seven days a week, for as long as it takes to recover.”

In hindsight, however, some of that overtime was not put to particularly good use.

“We spent an awful lot of time trying to test all our computers and validate them and make sure they were clean and safe to put back on the network,” James said.

“After about two weeks of doing it and redoing it and redoing it, we made the decision in the end just to wipe everything and start afresh.

“In hindsight I would have done that at the beginning and not wasted all that time and effort.

“We’d been hit by something very serious and I think it was not the best use of our time to spend trying to check all of that equipment and see if there was anything that was salvageable.”

General counsel Amber Matthews said one of the “saving graces” for DLA Piper was that the company did not lose any data to the attackers, and that its backups were unaffected.

Still, the company is making changes to its architecture to prevent a similarly catastrophic global failure should it be hit again in future.

In addition to segmenting its network so it can better contain threats, the company is also looking to stand up cloud-based versions of its core systems for business continuity purposes.

“We manage all of our infrastructure on-prem,” James said.

“But for core services we are now looking to host some of those services - at least as a lifeboat solution - in the cloud where we can failover to those very quickly if we need to.

“The assumption being this will probably happen again at some point, somehow, hopefully not on the same scale, but we can’t wait four days to recover email - we need to be able to fail that over almost instantly.”