Melbourne Uni adds 1.6PB of software-defined storage

The University of Melbourne is entering the final stages of an implementation of around 1.6 petabytes of distributed software-defined storage on commodity hardware for researchers across Victoria, using an open source platform called Gluster.

The large-scale storage refresh project kicked off in October, with the capacity to be made available to researchers across the state through the VicNode partnership with Monash University.

University of Melbourne principal advisor of research computation strategy Steven Manos told iTnews the university has progressively been moving away from high-end disk and NetApp solutions in favour of software-defined storage on commodity hardware using Gluster.

“For us at the University of Melbourne, software-defined networking is the way forward," Manos said.

"Having a platform on which we can deploy new types of storage services without having to necessarily purchase entirely new infrastructure is something that's very important to us."

University of Melbourne system administrators team leader Linh Vu said the storage upgrade is a response to growing file sizes used in academic research projects.

“In the past, VicNode had a limit of 100 terabytes per volume, but it became very limiting as some research projects progressed well past that 100 terabyte mark,” Vu said.

According to Vu, the large files meant a distributed file system optimised to handle sequential I/O on a large scale was required.

“In the past, if you wanted to buy a storage server, you’d buy a commodity box and install NFS, kernel daemon, or Samba daemon. That is OK if you have a small group of users, and everything is stored on that one box,” Vu said.

In the Gluster platform’s file system, known as GlusterFS, each server exports its storage via what are known as ‘bricks’. The platform then assembles these ‘bricks’ into a volume that spans across all the participating nodes, making it appear as if it’s a single storage device.

Vu said Gluster's architecture enabled high performance because loads are spread across the nodes; much larger storage volumes as nodes can be combined; and higher resilience.

Vu said Gluster was chosen both for performance and price reasons.

“The price point [meant] we ended up buying well over the amount [of storage] we originally wanted," he said.

"We set out for around 800 terabytes or so, and with the latest addition, we’ll have 1.6 or 1.7 petabytes of storage."

Vu said that Gluster is also outperforming a separate cluster running more expensive technology.

“Instead of putting more money into the expensive cluster we had, we’re investing more into Gluster to put more nodes in.”

Implementation

The deployment was designed with the assistance of a storage architect from Red Hat. Vu said the choice of vendor was made in order to leverage an existing agreement.

“For the operating system, you can use Gluster on pretty much any Linux build, but in our case we’re chosen to go with Red Hat Enterprise Linux because we’ve got a subscription to it. We also subscribed to Red Hat Gluster for initial support, because it’s the first time we’ve used it,” Vu said.

“Once designed, I began translating the specs for each node. We had a data centre team that helped us to install and network the whole thing, and then my team came in and built it.”

Vu said that while participating nodes on Gluster can be built using almost any kind of hardware, he opted for a standard architecture to make maintenance easier.

“The hardware we picked is a standard commodity head server, a Dell R630, with a PowerVault as a patch. The PowerVault can take up to 60 3.5-inch drives,” Vu said.

“We’re using 49, so that’s four by 12 drives, each of those 12 drives is a RAID6 group, with one drive as a global hot spare. And that’s pretty common for those kinds of storage boxes.”

According to Vu, most of the time was spent testing the configuration to make sure it would handle the workloads, then building the NFS and Samba cluster on top.

Once the hardware was installed, Vu and his team then built two different storage clusters on top of GlusterFS. The first is an NFS cluster that runs on NFS-Ganesha, Pacemaker and CoroSync, while the second uses clustered Samba platform called CTDB.

Challenges

While there are many benefits to using Gluster as a storage platform, Vu cautions that there can be a steep learning curve during the setup phase.

“There’s lots of small things you have to learn, like how the Gluster clusters work, how the NFS-Ganesha cluster works, how the NFS-Ganesha clusters work, how the CTDB Samba clusters work. They all have their little quirks,” Vu said.

“And because our use case is a little different, while it’s generally pretty mature, the documentation has a few little things missing. If we'd followed the documentation exactly it wouldn’t quite work, and so we needed to understand the technology and write custom scripts to make life easier.

“It takes a bit more time because of that, but we’re mostly there."

For anyone looking to deploy Gluster, Vu strongly recommends having a team in place who have a strong grasp of Linux administration.

“It’s not something where you can quickly read the documentation and build a storage cluster in a day. I know one of the common things online is building a cluster in an hour – that won’t happen. It takes a lot of careful planning to build something like this,” Vu said.

“If you do have people with the right skills, it turns out to be easier to build than our previous storage clusters. The benefits are in price and performance.

“But you need to look into the architecture, and one of the critical things in a storage cluster like this is you have to really limit the use cases and make them very clear.”