Scientists rein in the commercial cloud

Computing researchers have developed software designed to manage large workloads on commercial cloud providers that also ensures the workloads avoid vendor lock-in.

Developed for the Large Hadron Collider experiment LHCb, the open source software framework, dubbed DIRAC, has already distributed 10 million computing tasks across more than 100 global LHCb sites.

Other DIRAC instances have been deployed to manage tasks across the Europe-Latin America GISELA grid, the Cherenkov Telescope Array and for the Belle/Belle II project.

According to DIRAC developer Ricardo Graciani, the framework integrates seamlessly with local, grid and Amazon cloud resources such that these resources can be accessed via a secure web portal.

The software executes researchers' applications in virtual machines, while controlling the use of resources - and thus hourly cost - as well as monitoring output.

"DIRAC takes care of making all [these] resources look the same so that you do not have to worry if your application is running on the cloud, on the grid or at some local cluster of a group that collaborates with you," said Graciani, a researcher at the University of Barcelona's Institute of Cosmos Sciences.

"Depending on the amount of other resources you have accessible and the amount of computing you need, you can decide to enable this module, or use it on its own, to boost the execution of your calculations."

University of Melbourne programmer Tom Fifield worked with Graciani and his team to implement DIRAC for the Belle/Belle II project, which used a mixture of local, grid and Amazon EC2 resources.

The Belle/Belle II project is investigating why matter is more commonly observed in nature than antimatter.

Between January and August, DIRAC processed 21TB of Belle data with 27.8 percent of computing power from Amazon, 38.2 percent from collaborators in Japan and 5.82 percent from local resources in Melbourne.

Fifield said implementation involved three to four months of "concept work" and "a couple of months" of coding.

With DIRAC now set up, Belle/Belle II researchers would now need only a week to move to or add another cloud provider, he said.

"The design of the DIRAC cloud module is totally generic," Graciani explained. "Although we have only tested so far, at large scale, Amazon EC2 services, our plans for the next months is to make a survey of other providers and test their functionality, scalability and cost.

"With this information the user will at any time decide to use services from a different provider if he has a better offer or combine several of them to get a huge instantaneous computing power."