Inside the DOE’s power-hungry supercomputer

How will these techniques be implemented in future systems?

Linked gallery: Inside the DOE’s power-hungry supercomputer

Pete: For future machines, there are a lot of ideas being thrown out there about how to optimise cooling of computers, including thermal energy storage tanks, combining heat and power plants, geothermal cooling and ice slurry technologies.

We're working with IBM right now about the design of these computers. The current generation of Blue Gene/P uses air-cooling. The next generation will be water-cooled: we'll bring the water directly into the rack and there will be pieces of tubing that touch the hot parts of the computer, and so we're much, much more efficient because we can pull the heat off directly.

For our next-generation machines, we are working with engineers from the University of Illinois on how to optimise chilled water temperatures for next-generation computing hardware, and how to maximise our free cooling period.

What are some other techniques that you're using to reduce the power consumption of Intrepid right now?

Jeff: One of the things we're doing now is modifying the consumption profile based on the power cost of electricity.

When you do something really computationally intensive, the computer actually draws more power. We're trying to understand the applications that need more power, and run those in patterns that will allow us to use less energy.

For example, if we know that certain applications really consume a lot of power, those are the kind of applications we'd want to run in the evening, when the electricity is cheaper.

The BlueGene/P can vary in power [requirements] by almost a factor of three from idle to running really complex math operations. That's a pretty big difference; so being able to predict those swings is a big area of research for us.

Are there any of Argonne's energy saving techniques that can be applied to enterprise computing?

Pete: In fact, almost everything that's been described are things that data centres are looking at and beginning to try out.

How many nines of reliability you need does have an impact on the kinds of things you can try and the kinds of things you can do. For example, one of the things we do here to save power is we don't have a battery backed-up power supply for the entire machine; we only have one for the disk.

We think of supercomputers as a sort of time machine. What we mean is that a lot of the technology that we develop here for supercomputers is what will be in a high-end server five years from now, and in your desktop several years later.

An example is the move from air-cooling to water-cooling. We already know that our next generation [supercomputing] machine is going to be completely water-cooled, because that makes the most sense from a green perspective.

In the future, we're going to see standard data centres slowly shifting over to water-cooled servers. It seems pretty likely to us that, as we continue to have larger compute requirements, designing these more efficiently will be what the industry does.

Another time-machine concept in the Blue Gene/P architecture is by going with a slightly slower clock, and increasing the number of processors, we save a lot of power. We're already starting to see that in data centres now, where choosing the fastest, most powerful computer for each individual node is not the best power envelope.

Choosing one that is slightly slower, but having more of them, uses less electrical power. Data centres like Google, Amazon and Yahoo can take advantage of having lots of servers running at relatively lower power and this is another trend that we expect will continue from our space into the data centre space.

Jeff: The free cooling thing is clearly dependent on geography, but the concept of optimising chilled water temperature based on your actual demand is something that all data centres should be doing.

It's really an integration of IT professionals with engineers; it's understanding how the applications are running and how hard they are pushing the machine.