CSIRO flags in-sourcing some of its AWS workloads

CSIRO will look to in-source workloads currently running in AWS to a powerful new private cloud it is building on OpenStack and Mellanox technologies.

The research organisation’s HPC technical lead Jacob Anders revealed to the OpenStack summit in Vancouver last week that it had already run a proof-of-concept for the new private cloud environment.

“We’ll be pushing forward to transition this PoC into production in the coming months,” he said.

Anders said that “significant investment” would be made in the new private cloud.

CSIRO already runs research workloads on a mix of compute, from traditional batch-queue HPC systems to cloud services running both on-premises and in third-party data centres.

“We’ve got about two petaflops of compute and 50PB of storage in-house,” Anders said.

“In the last few years, we’ve see increasing demand from [researchers] for custom and dedicated HPC services or systems that are configured in a significantly different way than our [traditional] shared systems.

“We also see growing interest in cloud. We have a significant [AWS] EC2 user base with a fair bit of growth there, so we’re thinking about possibly in-sourcing some of those workloads.”

Anders said the company saw high-performance private cloud as a platform that could meet the performance expectations of researchers, while also providing the organisation with more flexibility than traditional HPC.

He said the organisation still planned to keep its HPC resources online.

“We’re a big HPC user,” he said.

“HPC provides us top performance and our users love it. They will never give it up, ever. It’s not an option.”

On the other hand, Anders noted the rise of cloud had shown CSIRO that there was value in having operational flexibility in the way research infrastructure functions.

“We wanted to find a solution that could provide both the performance of HPC and the flexibility of cloud,” Anders said.

“This is something that would really help us and have a lot of value for the organisation.

“The good news is we found a solution that can deliver us the top performance that we’re used to from batch-queue systems and also that flexibility we’re seeking that’s specific to the cloud world.”

The new public cloud will run on an Ironic-based instance of OpenStack that is supplemented with software-defined Infiniband, Anders said.

Ironic is the bare metal provisioning component within OpenStack. Bare metal provisioning means workloads run directly on hardware rather than inside of a virtual machine.

Anders said he believed that, once stood up, the new private cloud would be “a real game changer” for CSIRO.

“We believe this will be a real game changer in how we work because this allows us to apply DevOps and infrastructure-as-code methodology to high performance computing, which gives us that flexibility and ability to make things dynamic, which we’ve been lacking,” he said.

“We also believe that this will pave the way for the next generation of lightweight, highly efficient scientific computing platforms and that’s a great asset to us.”