Atlassian thwarts bitcoin mining attack on Kubernetes environment

Atlassian has revealed how it stopped a bad actor from exploiting free build minutes in its Bitbucket Pipelines service to mine cryptocurrency.

The Australian software maker did not say when the attempted bitcoin mining abuse incident occurred, but indicated that it had been able to contain the attack within 15 minutes of it starting.

Bitbucket Pipelines is an integrated continuous integration and continuous delivery (CI/CD) service within the Bitbucket code repository that provides a way to move code from development into production.

New builds run inside Docker containers on Atlassian’s Kubernetes-based infrastructure. Bitbucket Pipelines has a free tier which includes 50 build minutes a month - machine time that the attacker tried to exploit.

“We offer Bitbucket Pipelines to the public where any developer hosting their code in Bitbucket can quickly set up a build pipeline from a simple YAML file to continuously integrate, build, and deploy their code,” Atlassian’s Kubernetes platform lead Corey Johnston said.

“While this is a fantastic service for developers, at the end of the day for platform administrators like my team, this represents arbitrary code execution because we’re giving users the power to do anything in these build environments.”

“Our Kubernetes clusters run arbitrary, user-provided code. In our Bitbucket Pipelines clusters, these users are non-Atlassians, and all of these customers’ code runs on the same pool of hardware inside a given cluster,” Atlassian senior engineer Matt Whittington explained.

“This configuration provides a big security risk for us to manage, it increases the surface area that attackers can probe and it increases the likelihood that any unpatched machine or other vulnerability can be exploited to gain unauthorised access to our systems or data.”

Johnston said that having arbitrary code running “on a multi-tenanted platform like Kubernetes means we take security very seriously.”

While Kubernetes enabled Atlassian’s platform administrators and security team “to define a baseline cluster security policy”, the company relied on a defence in-depth approach to secure its 20 Kubernetes clusters, layering protections and leaning on technology from both AWS and Tigera.

It was the Tigera Calico technology that Atlassian partially credited with helping to cut off the crypto-mining attack.

“One day, the Pipelines team paged us into an incident room and alerted us to some potential bitcoin mining happening inside their cluster,” Whittington said.

A bad actor had “covertly” started to use Bitbucket resources to mine cryptocurrency “by spinning up new instances on Bitbucket and exploiting free build minutes”, Atlassian said in a slide deck.

“In a matter of minutes, we identified the incident and deployed a global policy [using Tigera Calico rules] that blocked all connections from containers across the cluster (approximately 40-50 nodes) to the specific IP addresses receiving the mined coins,” Atlassian stated.

Whittington said that Atlassian first verified that the Calico rules blocked the expected traffic in a development cluster, before applying the rules to the production cluster via the Kubernetes API “and halting the abuse immediately.”

“This is a big win because without Tigera we may have had to script a solution to the problem dynamically and go into each node manually. This would likely be more error-prone and much slower.”

Whittington said Tigera would play a part in two more initiatives.

First, it would have a role in securing the company’s next-generation platform-as-a-service (PaaS), which is currently being worked on.

“Powered by Kubernetes this PaaS will enable every developer in Atlassian to build and ship faster,” Whittington said.

“Tigera’s helping us address application layer policy requirements for this nextgen PaaS.”

According to Whittington, a second use case for Tigera is around adoption of Windows containers.

“With Windows containers gaining more and more momentum in the container ecosystem, we’re also keen to support running Windows pods in our Kubernetes clusters,” he said.

“The main use case for this is that some Atlassian teams would love to be able to use the same CI/CD pipeline and tooling they already use to build Windows binaries with our apps.”