Telstra has overhauled the system its cyber response team uses to gather and store forensic evidence following a security incident in its own - or in business customers' - environments.
Morgan Arundell, a security incident senior analyst in Telstra’s cyber security operations team, told the recent AWS Summit in Sydney that the telco redefined its forensics acquisition strategy in 2017, and implemented the strategy via a new cloud-based forensics system last year.
The new strategy “meant responding faster at greater scale than ever before to be able to minimise the impact of security incidents in our environment - and of course, to do it with fewer resources and at a lower cost,” Arundell said.
“For me personally, that meant automating the boring parts of my job, so I could focus on the more interesting fun work,” he said.
The “boring” parts of the job were around acquiring data from targeted systems and drawing that back to a central point where it could then be investigated.
The hope was that security analysts could reclaim time spent on administration and invest that time instead into actual investigative work, where they could add the most value.
Redefining response
Arundell said Telstra wanted to achieve “the fastest possible restoration of normal business service” following a security incident.
That meant "putting all of the information that our analysts need in their hands in one shot so that they can quickly triage and make decisions about an incident and what needs to happen next,” he said.
The telco also wanted to better preserve the integrity of evidence it collected as part of its own internal investigations, in case that evidence was then needed for a court case.
“For forensics, we're focused on ensuring integrity of the evidence for presentation in a court of law,” Arundell said.
“That means preserving the chain of custody regardless of whether we're looking at physical collections or digital acquisitions, maintaining the integrity of the evidence throughout the evidence lifecycle, and maintaining an accurate audit trail.”
“Unfortunately, when an incident kicks off, we're often not sure just how time sensitive that incident is going to be, or whether or not we'll need to present evidence in a court of law. So we need to be able to meet all of those criteria all of the time.”
Implementing these requirements meant moving away from an on-premises system the team had been using, and instead standing up a new system to work out what to capture about an incident, and then to capture it.
Cloud forensics
According to Arundell, the new system had to “firstly, identify that acquisition is required and the scope of that acquisition; isolate the resources in question; acquire and store the data; and to process it into a form that's usable for analysts.”
From there, analysts like Arundell could interrogate and investigate the evidence that had been pulled together.
“It's that last step where you really want your analysts to be spending the majority of their time because that's where their real value-add is,” he said.
Data acquisition had to be possible from on-premises and cloud-hosted systems, as well as from customers’ systems (the latter being a security managed service sold by Telstra).
In particular, Arundell’s team wanted to simplify data acquisition and forensics on incidents impacting the growing number of cloud services used inside Telstra.
“Many of the processes that we had in place for acquiring data across multiple cloud providers were highly manual, which meant that we had lead times in acquiring data and then analysing it, which resulted in a slower time-to-respond for some of our security incidents,” he said.
“On top of that, our cloud workload was also increasing. Telstra has been rapidly increasing its adoption of cloud services over the last few years.
“The attacks we were seeing were also becoming more complex and generally sneakier so we needed to go deeper than ever before to be able to ensure that we could see the whole picture.”
Arundell said Telstra “decided that we needed to build an innovative, scalable platform that would allow us to do forensic acquisitions globally across hundreds of Amazon accounts, multiple subsidiaries and across multiple legal jurisdictions.”
Initial uplift
The telco brought AWS and the cloud provider’s professional services arm onboard to help with the project.
It was able to use a conceptual design by Ben Potter, AWS’ security lead for the Well-Architected framework, a measure of current best practice in the AWS ecosystem.
“We decided that we needed to establish three initial capabilities that would target for us as a starting point,” Arundell said.
“The first was AWS-to-AWS acquisitions - acquisitions of disk and memory from EC2 instances in Amazon.
“The second capability was an on-premises to AWS capability. Telstra operates in a hybrid environment with multiple cloud providers, over 150 distinct on-premises networks, multiple subsidiaries, on a global scale, in multiple legal jurisdictions.
“Many of the networks that we operate in have limited direct internet or AWS connectivity, and that's by design. We don't have an intent to change some of those networks, so we needed a capability to be able to acquire data from across all of those networks and push it back to AWS for further analysis.
“Finally, [we wanted] a capability for doing third party acquisitions. That would enable us to interact with third parties like Telstra partners and managed security services customers.
“All of these capabilities would push data back into a single backend processing facility for storage, processing and presentation to our analysts.”
The core of the new incident response and forensics system is built on an AWS stack.
The team has also “started to automate some of our common investigative processes”, Arundell said, though he noted the automation is enabled using “a mix of open source and proprietary tools.”
Next steps
With the core system bedded down, Arundell said that work is now underway to integrate the forensics system “with other Telstra systems, particularly our case management tools.”
Telstra is also augmenting the system to improve forensics capabilities around containerised workloads.
“We’re building out a solution for container-based forensics for targeting platforms like ECS [Elastic Container Service], EKS [Elastic Kubernetes Service] and Fargate, that will allow us to also perform scalable triage across the entire Telstra fleet.
“That's built using the open source GRR Rapid Response project, and an AWS Fargate and Aurora backend,” Arundell said, “But that's a talk for another day.”
Overall, Arundell said that the incident response and forensics uplift had enabled Telstra’s cyber response team “to increase our acquisition and processing speeds, reducing acquisition and processing times from days down to hours.”
