A hacked Australian Defence subcontractor lost 30GB of "commercially sensitive" documents on projects including the Joint Strike Fighter (JSF) program and the P-8 Poseidon “submarine killer” plane, as well as detailed designs of Australian Navy ships.
The government only disclosed the hack on Tuesday and has so far offered scant details.
But the Australian Signals Directorate (ASD) yesterday revealed a far more detailed post-mortem of its investigation into the hack in a presentation to the AISA national conference in Sydney.
ASD incident response manager Mitchell Clarke revealed that the attacker managed to gain “full, unfettered access to the environment” of the victim.
It is believed the attacker was “an APT [advanced persistent threat] group or nation state group”.
The ASD dubbed the attacker “APT Alf” after the Alf Stewart character in Australian soap opera Home & Away. “He’s just an angry dude,” Clarke said.
The victim was described as an Australian aerospace engineering firm “four levels of subcontracting down” from primary contractors to local and US Defence agencies, including Boeing and Lockheed Martin.
It was unclear exactly who the subcontractor had been working for.
The company had been vetted to work on US military projects through a scheme known as International Traffic in Arms Regulations (ITAR), though Clarke noted the vetting process is not particularly thorough.
Most of the 30GB of data that APT Alf managed to exfiltrate related to high-profile allied Defence projects.
The data included sensitive details of the Joint Strike Fighter (JSF) project being pursued by the US and its allies, including Australia.
The hacker also gained access to details of Lockheed’s C-130 planes, the Boeing P-8 Poseidon plane which is used for “long-range anti-submarine warfare”, Boeing’s Joint Direct Attack Munition (JDAM) smart bombs, and of “a few Australian naval vessels”.
“We found one document that was like a wire diagram of one of the Navy’s new ships,” Clarke said.
“You could zoom in down to the captain’s chair and see that it’s one metre away from the [navigator’s] chair – all very good exfil for the actor.”
What follows is a detailed account of what is known about the breach, from the perspective of the ASD who – along with CERT Australia – was jointly involved in forensic investigation of the hack, and in helping the victim to secure its network.
Disbelief
Both ASD and CERT Australia were tipped off to the incident by an undisclosed “partner organisation” at the start of November 2016, though the actual infiltration happened in July that year.
“The partner knew about the activity in July; it just took them a long time to go through the legal and regulatory processes to tell us,” Clarke said.
“[That’s] not the best outcome for us, the victim or Australia as a whole, but at least we were told. It’s much better than not being told.”
Both organisations attended the victim’s offices as soon as they received clearance from the partner.
The victim was wary when the ASD and CERT Australia showed up on their doorstep one morning.
“They didn’t believe who we were because we had no credentials to show them,” Clarke said.
“We just told them ‘We’re from Canberra, it’s ASD and CERT Australia, we’d like to talk to you about a problem’.
“[They] called both the ASD incident response hotline and CERT Australia hotline, asked if we worked there and if we were coming to meet them today, and both hotlines said, ‘We don’t know who you are, no one’s going to come to talk to you, it’s definitely a scam’.”
Clarke didn’t say how the impasse was resolved, but it was.
That enabled the ASD to take some initial actions, deploying host-based agents to as many endpoints in the victim’s environment as possible.
ASD used the open source Google Rapid Response (GRR) to pull historical logs and artefacts, and a subset of Carbon Black to identify and record process activities like “module loads, network connection creation events, registry modification events, and file modifications”.
“We’re just trying to identify and start recording everything that’s happening in that environment,” he said.
“If we had an agent that combined GRR and Carbon Black’s functionality, we’d be gold. But it doesn't exist.”
Clarke’s team also sourced a year’s worth of monthly back-ups for all servers in the contractor’s domain, which was important as ASD was coming in months after the attack first began.
Gain entry, fan out
APT Alf’s entry point to the contractor’s network was via an internet-facing IT helpdesk server running outdated software that contained an “arbitrary file upload vulnerability”.
“All the actor did was exploit the vulnerability and upload a web archive file,” Clarke said.
The uploaded file contained a copy of a backdoor known as the China Chopper Web Shell.
Once in place, the actor used it to “upload a whole bunch of other tools – network enumeration tools, some DIY scripts, lateral movement tools and cred dumping tools”. The cred dumping tool is thought to be a modified version of Mimikatz.
Using the Mimikatz variant, the actor was able to obtain all login credentials cached on the IT helpdesk server “as well as all local credentials”, Clarke said.
The local admin credentials were particularly valuable as the victim company had set up “common local administrator accounts on every machine in the organisation”.
“What I mean by common is it’s got the same username and same password on every single host,” Clarke said.
“It’s not a domain account; it’s a local admin account for that machine.”
Clarke said it was a configuration that ASD incident response saw “all the time” – but one he recommended disabling on all enterprise machines and servers.
“I think it’s a hangover from late '90s / early '00s systems administration where your helpdesk staff or management staff needed a local admin account on every single host so they could log in for testing purposes or helpdesk purposes when the domain falls down,” Clarke said.
“We as an industry need to stop doing this. Local admin accounts cannot be common across an organisation.
“Once the actor obtained that local admin account [on the IT helpdesk server] they got local admin on every single Windows host within the whole environment.”
Local admin credentials enabled the actor to move from the helpdesk server laterally into other parts of the network.
From the helpdesk server, ASD tracked the actor’s movement “straight to the domain controller … to gather more credentials”.
The tracking was made possible because the actor’s tool crashed the domain controller, creating a process dump that the ASD was able to examine.
“Nine hours after they dumped creds from the domain controllers, LSASS [Local Security Authority Subsystem Service] on the domain controller crashed,” Clarke said.
LSASS verifies users logging into a Windows machine, handles password changes and creates access tokens.
“When the actor’s tool injected Mimikatz into LSASS, it created some sort of instability in the process,” Clarke said.
“Nine hours later, the whole process died and the machine blue-screened and fell over. Every time that happens you get a process dump of the offending process stored by Windows, which is awesome for forensics.
“Inside that process crash dump we actually see a Mimikatz-like output from the actor’s cred-dumping tool.
“From here we know that the actor has obtained all usernames within the environment as well as the hashes for all those users.
“Then it’s either a case of parse the hash, which we found historical evidence of them doing where they could, but for stuff they couldn’t, they’ve taken the hash away to crack it offline.
“Some of the privileged accounts didn’t have complex passwords so the hash cracking was relatively easy.”
At this point, Clarke said, the attacker had “full unfettered access to the environment”.
“We saw the actor log on to pretty much every server within this environment,” he said.
The actor rifled through an engineering document store and even some of the company’s email accounts, though ASD was unable to work out what emails they had specifically accessed.
“We identified the actor was logging into the OWA – the Outlook Web Access server – with legitimate credentials to read emails,” Clarke said.
“They were reading the chief engineer’s emails, the finance person’s emails and one of the contracting engineer’s emails. We [just] couldn’t work out which ones.”
The exfiltration
Having established the initial point of access and how the attacker moved around the internal network, ASD then sought to establish what was stolen, how and from where.
The partner organisation that provided the tip-off on the breach also shared some additional details: that 8GB of encrypted data had been exfiltrated from what appeared to be a remote desktop server.
However, an examination of the remote desktop server and service turned up “no evidence of abuse”.
The ASD did find two things of interest on the server hardware: a China Chopper Web Shell and a copy of the WinRAR data compression tool for Windows “that was dropped about the same time as the partner told us about the exfil”.
“Doing memory forensics on this box, which had been running since the exfil, we got super lucky with a RAR command that was still sitting in-memory,” Clarke said.
“The server had about 16GB of RAM and was only used [for remote desktop access].
“Not a lot was happening in its memory, so we were lucky that this [RAR] artefact hadn’t been clobbered yet.”
The RAR command revealed the actor’s plans to encrypt and compress stolen data into “8GB of exfil”, which would be hosted as two PNG [picture] files, each 4GB in size.
“The really nice thing is this gave us the source of the data. It showed us that the data came from the organisation’s SharePoint instance,” Clarke said.
“This doesn’t happen a lot of the time for us. A lot of the time we’ll come away from an engagement not knowing what the exfil was or even if it happened at all.
“We assume it always happens or lateral movement to a partner organisation has occurred and that was the goal of the actor, so for us to find this artefact was super lucky.”
The directory that was compressed into the RAR archive actually contained about 30GB of Defence and other commercially sensitive data.
"The way they downloaded it was just to host it or place those two 4GB PNG files into a web-accessible directory," Clarke said.
"Then the actor just pointed a download manager at those two files and downloaded them via HTTPS. There’s no amazing tradecraft [involved]."
The exfiltrated data affected a number of projects regulated by the US’ ITAR scheme, as well as some Australian Navy vessels.
Remediation works
Having established the point of access, how the actor moved within the network and what they had been able to exfiltrate, ASD moved onto remediation works with the victim.
“We always recommend organisations take no action during the investigation phase,” Clarke said.
“That’s really hard for people to accept because the natural human response when you discover a threat is to remove it, but if you did that, you’d tip [off] the actor and potentially lose the ability to understand the rest of the breach.
“If you can give us time to understand the breach you’ll spend the least amount of dollars in remediation and if we charged investigation costs, and you’ll have more confidence you’ve successfully removed the actor’s access to your network.”
In the first instance, remediation meant doing “just enough to make sure these guys can’t just jump back in”.
For the contractor, it meant taking basically their entire IT environment offline, improving the security posture and patching holes.
Even after this point, Clarke said that ASD continued to monitor the environment “as if we’ve missed something. We assume they’re still pwned,” he said.
The final phase of remediation focuses on long-term system hygiene – and the contractor had some big lessons to take from the incident given that much of its problems stemmed from what Clarke called “sloppy admin”.
No need for a witch-hunt
Clarke defines sloppy admin primarily in terms of the Defence contractor’s decision to have “one IT person for the whole organisation”.
The current IT manager has been in the role for just nine months.
“I’m not throwing stones on this one IT person [over the breach] because there’s no way this one IT person could have done everything perfectly across the whole domain,” Clarke said.
Clarke said that ASD had learned the hard way that the best approach to incident response was to start by bringing executives, IT and infosec staff of the victim company into a single room, tell them they’ve been compromised but that “it’s no one’s fault”.
“The less mature an organisation is, the more they want to blame someone,” he said.
“That’s what I’ve noticed from doing this the past few years.
“But [you have to explain to] the execs and IT staff that ‘Hey, you guys held valuable data and sure there’ll be a security vulnerability, flaw or misconfiguration in your environment that led to this breach, but the fact you’ve been breached is probably going to happen anyway, and there’s no need to go on a witch-hunt’.
“Let’s not go after individuals but let’s treat the problem and solve it.”
Clarke saw no benefit in shaming any one party over a breach. Rather, he was resigned to what is fast becoming an inevitability for many organisations.
“If you’ve got data of value you’re going to be popped at some stage,” he said.
“Let’s be a little bit adult about this and just get on with it.”
