Research network builds storage cloud

Australia's Academic Research Network (AARNet) has launched a file transfer service making use of cloud storage and is using open source tools to code an abstraction (virtualisation) layer to enable the service to burst onto other storage clouds should additional capacity be required.

The service, called CloudStor, allows AARNet's users - many of whom routinely send high resolution imagery or other large files across the network - to more easily transfer these large sets of data by first uploading to one of AARNet's highly-tuned Storage Area Networks (SAN).

Files transferred using CloudStor are temporarily stored in a 4TB partition of this 24TB SAN.

Guido Aben, Director of eResearch at AARNet told iTnews that traditionally, when academics needed to transfer large files, they were either delivered physically or via FTP (file transfer protocol).

But today, "fewer and fewer sites run their own public access FTP server and peer to peer transfers are increasingly looked upon with suspicion," he said.

CloudStor, deployed on VMware virtual machines and running on a Red Hat Enterprise Linux server, is a "central server on AARNet that customers can trust" for a more efficient transfer of files across the network, he said.

"The CloudStor service looks like a web file transfer, it [transfers] over port 80, but it actually sends the files to our server and patches into whatever storage we have available."

 Bursting to Amazon's storage cloud

AARNet engineers are working on an overflow feature - a storage abstraction layer that would enable its storage cloud to burst onto third party providers when the space allocated for CloudStor on AARNet's SAN is exhausted.

Should the 4TB partition fill, the system fails over to the next available storage unit within AARNet's IT infrastructure, after which it will burst onto Amazon.com's S3 cloud storage service.

This 'overflow feature', currently in proof-of-concept testing, will most likely include an extension module built using FUSE (File system in User space), a freely available set of open source code that allows a developer to create a new file system without editing the kernel code of the operating system.

FUSE is particularly useful for developing virtualised applications. AARNet engineers are using it to ensure that CloudStor's file system doesn't write to a specified unit of storage, but rather provides a translation or path to whichever virtualised storage unit contains the data.

Aben said that the FUSE project already boasts plug-ins to Amazon.com's S3 cloud storage service.

He also said that while AARNet's users are unlikely to need to burst onto a third party cloud storage service like Amazon, the bursting feature was required by some of AARNet's partner networks in Norway and Ireland.

The academic research networks in both of these countries conduct joint development of services such as CloudStor with AARNet, but neither have the need to build out large SANs as AARNet has.

"The Norwegians made it very clear to us, when we implement storage failover, the system needs to be able to write to the cloud as well," Aben said.