Garvan Institute revamps HPC cluster

Sydney’s Garvan Institute for Medical Research has revealed efforts to revamp its Dell EMC high performance computing (HPC) cluster as it probes deeper into the human genome and the tantalising possibilities of personalised medicine.

Garvan's research lab

“When you think about it, every disease comes down to DNA,” said Garvan’s chief of informatics Dr Warren Kaplan at the Dell Technologies Forum in Sydney on Tuesday.

This includes genetic illnesses, how viruses or bacteria interact with their hosts, or even how individuals respond differently to medicines.

And in order to unravel the secrets in the double helix, you need a lot of compute power.

The first time scientists sequenced the human genome it took US$3 billion and 10 years. Now Garvan researchers churn through about 50 people’s entire genetic sequences per day at a cost of US$1000 a hit.

The Data Intensive Computer Engineering (DICE) group at Garvan alone has helped the institute process more than 10,000 whole human genomes - each containing 5 million points of differentiation that Kaplan says amounts to “a quarter terabyte”.

Kaplan initially built a scalable platform called Vectis, an open-source genome analysis platform for biologists, clinicians, researchers and data scientists to store, analyse and query the plethora of data at hand.

The institute then approached Dell EMC to build their high-performance computing capacity in 2012. While the original compute and data storage facilities will continue to operate 24/7 well into the future, work is already underway to upgrade Garvan’s on-site data centre and HPC.

Operations to secure the Garvan data centre’s burgeoning electricity supply and cool requirements began in June this year, This is to make way for the 47, 14th-generation PowerEdge servers and NVIDIA GPU array.

Dell EMC’s Australian head of HPC Andrew Underwood said the institute will only have 24 GPU cores to undertake deep learning and other processing-intensive operations.

Kaplan explained that this is because the field of genomic informatics is evolving so rapidly nobody’s quite sure if a significant investment in GPUs will stand the test of time.

Garvan itself has gone from sharing one informatician (Kaplan) with the Chang Cardiac Institute in 2002 to now having over 80 staff dedicated to informatics projects.

So instead of putting all their eggs in the GPU basket, Dell is integrating 10 Intel Arria 10 GX field-programmable gate arrays (FPGA).

Underwood said FPGA technologies have been around for decades, featuring in everything from medical devices to mission control panels, because of their abilities to undertake a specific task.

While other FPGAs have recently been used for AI projects overseas, Underwood said this the first time the devices have been deployed for such uses in Australia. Intel believes it to be a first in the Asia Pacific region.

“These FPGAs have become so commoditised that they can now be put into a common compute platform. The advantage these provide over a traditional CPU is that, if you think of a CPU processor, it might have up to 28 compute cores and each core can undertake a task in serial.

“But an FPGA is designed for much higher throughput and for parallelisation of data. So on a single FPGA device, we can see that they can move data up to 30 times faster than a traditional central processing unit,” Underwood said.

“What that means is that larger datasets can be processed faster, potentially at lower latency.”

He said that with the recent delivery of the FPGAs, the Garvan Institute could soon be moving it’s genomic work onto those arrays and adopting new sequencing techniques that are too intensive for CPUs.

Other upgrades include:

• 122,880 NVIDIA Tesla V100 CUDA cores
• 15,360 NVIDIA Tensor cores to accelerate HPC and AI techniques
• 744TB of NVMe to accelerate in-memory processing of genome datasets
• 41TB DDR4 RAM Memory
• 530TB usable capacity CephFS storage

The increased capabilities to sequence and analyse genomes will prove particularly vital to precision cancer treatment initiatives at Garvan, including the Lions Kids Cancer Genome Project and the Australian Genomic Cancer Medicine Program.

The latter seeks to uncover new treatment options for patients across Australia who have rare cancers or have exhausted other treatment options.

It compares the patient’s DNA code with the genome of the tumour in order to discover the underlying cause of the cancer and better target treatment accordingly.

The federal government recently announced a $50 million grant to bring the program to every state and territory in Australia.