Photos: Inside Animal Logic's latest server cluster

This week Alex Timbs and the IT team at Sydney-based animation house Animal Logic attended the opening of 'Legend of the Guardians: The Owls of Ga'Hoole", the latest feature-length animated movie to be built on advances in the computer processing power.

The opening marked the end of a three year effort for Animal Logic's IT team, which built and managed a massive cluster of HP blade servers to render the film from co-location racks housed within Sydney's Equinix datacentre.

The cluster of G5 and G6 blade servers, at its peak, had a core count of 11,336 - close to twice the core count of the supercomputer built to render Happy Feet II.

Alex Timbs, Head of IT at Animal Logic told iTnews that the animation house began with 500 blade servers but had to double the size of the cluster mid-way through the project.

"You can make assumptions on how much capacity you'll need based on the technical requirements of the movie, set against the experience of making movies in the past," he said.

But the total headroom required isn't 100 percent predictable, he said.

"Thankfully, you usually get a sense of needing extra capacity coming from a few months out."

A future cloud service?

The expansion took around one week to provision. The base installation, by contrast - which included imaging of systems and networking - took between two to four weeks.

"We're getting pretty good at it these days," Timbs said.

But Timbs said that as the cloud computing market matures, filmmakers may look elsewhere for extra compute capacity during peaks in processing requirements.

"We are interested in leveraging other compute capacity out there," he said. "I am watching [cloud computing], personally, very closely.

"The whole [film] industry would be interested in render-on-demand or render-as-a-service for those peak cycles in the film making business. But we haven't seen a financial model that works for us yet."

Timbs said excess compute supply tends to exist in territories too far away to ship large volumes of data to.  

"It comes down to the cost of getting networked to those locations that have capacity," he said. "We don't have lots of clusters with excess capacity in Australia."

There are plenty in America, he said, but Animal Logic's data sets are so large that transporting the data is prohibitively expensive.

"America doesn't make sense, but dark fibre to Sydney and Melbourne might make sense."

Render smarts

Harnessing the power of the 11,000-odd compute nodes requires farm management or "render scheduling" software, which categorises and breaks up rendering tasks into consumable chunks to be distributed to the various nodes, based on available capacity and priority.

Animal Logic uses internally developed proprietary software for this task, which Timbs described as "incredibly advanced" compared to commercially available products.

"I would say it gives us a competitive edge," he said. "Relative to the complexity of this film - which has the most expansive effects you could imagine - we would have needed much more than 11,000 cores without it."

This film required three times the processing and memory of the original Happy Feet, he said.

New film, new build

Timbs said his team felt incredible pride watching the finished film.

"Every one of my staff were elated - delighted, to watch the film.

"We do feel very much a part of it," he said. "The film is made on the infrastructure we provide."

The cluster has since been used for some one-off video effects projects and the main core of 500+ blades will be kept in-house for use in TV commercials and other smaller projects.

But the cluster is already at an age where it was unlikely to be used for Animal Logic's next blockbuster. Timbs said Animal Logic's next animated film was going to need the next generation of hardware to create.

"These films take three years to make," he said. "The memory and processing requirements for these films - as people expect more photo-realistic effects and animation - are going up exponentially. Meanwhile, the hardware every two years will offer three times the number of cores than the last generation. So we have to refresh hardware for every major project."