NEWS
GOVERNMENT
SECURITY
AI
DIGITAL NATION
RESOURCES
PODCAST
EVENTS
ADVERTISE

BUSINESS CLOUD DATA CENTRE

EDUCATION FINANCIAL SERVICES HARDWARE

HEALTHCARE INDUSTRIAL NETWORKING

PROJECTS SOFTWARE STORAGE

STRATEGY TECHNOLOGY TELCO/ISP

HR MARKETING FINANCE LEADERSHIP GOVERNANCE EMERGING TECH

STATE OF SECURITY 2025 State of HR Tech State of AI 2024

Special Reports Focal Points MEDIA HUB

PARTNER CONTENT PARTNER HUBS RESEARCH

Data&AI Breakfast SECURITY AWARDS DIGITAL NATION AWARDS BENCHMARK AWARDS Digital As Usual

NEWS

BUSINESS CLOUD DATA CENTRE EDUCATION FINANCIAL SERVICES HARDWARE HEALTHCARE INDUSTRIAL NETWORKING PROJECTS SOFTWARE STORAGE STRATEGY TECHNOLOGY TELCO/ISP

GOVERNMENT SECURITY AI DIGITAL NATION

HR MARKETING FINANCE LEADERSHIP GOVERNANCE EMERGING TECH

RESOURCES

STATE OF SECURITY 2025 State of HR Tech State of AI 2024 Special Reports Focal Points MEDIA HUB PARTNER CONTENT PARTNER HUBS RESEARCH

PODCAST EVENTS

Data&AI Breakfast SECURITY AWARDS DIGITAL NATION AWARDS BENCHMARK AWARDS Digital As Usual

ADVERTISE

Log In

Email:

Password:

Remember me | Forgot password?

Don't have an account? Register now!

Home
Galleries
Technology
Hardware

Photos: The data centre behind CERN's LHC

Inside Tier Zero.

on Aug 3 2012 7:00AM

CERN's 'Tier Zero' data centre is located on the same site as the Large Hadron Collider, a high-energy particle accelerator set in a 27km tunnel between Switzerland and France, near Geneva. The €7.5 billion project aims to prove or expand upon several key theories in the field of particle physics.

Six detectors - each operating discrete experiments - are housed within the LHC. Each operates its own server farm on site, for real-time filtering of results from particle beam collisions occurring every 25 nanoseconds. From there, results that appear to be of interest are sent in close to real-time to the CERN data centre.

The CERN data centre, or 'Tier Zero', was built in the 1970s. Its high ceilings once accommodated mainframes and a conveyer belt delivering rolls of magnetic tape between two floors.

Today, it forms the central hub of the Worldwide Large Hadron Collider Computing Grid, which harnesses the power of some 160+ computing facilities around the globe.

The lobby of the data centre includes a Google Earth map of jobs sent from CERN's Geneva data centre (Tier Zero) to about a dozen Tier One data centres (for near real-time processing of data) and over 150 Tier Two data centres (for batch analysis).

Some 1.5 million jobs are sent per day. Even when the LHC isn’t in operation, analysts are requesting access to older data to test new theories.

The core of Tier Zero's contribution to the Worldwide Large Hadron Collider Computing Grid is rows and rows of disk and CPUs. It is built almost entirely out of commodity computing products rather than highly sophisticated supercomputers. The middleware not only distributes jobs but also records what experiments have used what compute resources for the purposes of chargeback.

The Worldwide Large Hadron Collider Computing Grid takes up most of the raised-floor style CERN data centre. A lot of plant (generators, telecoms equipment) is housed externally, with UPS stored in a basement room.

The CERN data centre houses more than 60,000 processing cores and roughly 60,000 disks.

The facility also contains about 600 10Gbps connections.

Every system (server) is connected at 1Gbps, and each is uploaded to a switch with an uplink at 10Gbps to a massive routing core (a farm of interconnect routers), which boasts 7-8 Tbps of switching capability.

CERN was one of the very first internet sites in Europe. CERN staff have been using the Internet since the early 80s when it was primarily an academic research network.

As such, Tier Zero became one of the first digital exchanges - this is where many ISPs put their equipment to connect with other ISPs. This head start had a snowball effect and Tier Zero remains a popular ISP exchange.

CERN is currently upgrading its core routing infrastructure to support synchronous connectivity with a new facility being built in Budapest, Hungary. Pictured here is part of the network backbone for the grid and the accelerator itself.

The lower floor was originally a tape library connected to the second floor via a conveyer belt. Operators on the upper floor would send down a call for a specific tape, operators in the lower floor would collect it from the racks and put it on the conveyer belt.

Today, this lower floor is primarily used for disk and tape storage.

CERN uses tape storage for long term archiving. It stores some 60 petabytes on tape at the CERN site.

The tape robot pictured here takes requests from servers upstairs, scoots up and down the tape machine scanning the bar codes attached to the spine of the tape cartridges and mounts the appropriate cartridge.

CERN currently uses three manufacturers for its tape arrays – IBM, StorageTek and SpectraLogic. "We don't believe in having one supplier for anything," said David Foster, deputy head of IT at CERN.

The lower floor also includes a Linux compute cloud for use by CERN staff and those of its collaborating partners. CERN employs 2250 staff, but the CERN IT team serves some 8000 users, including students and staff working on-site at LHC experiments.

As the price of disk has dropped over time, CERN now runs at least 20 petabytes of disk on-site at Tier Zero (vs 60 petabytes of tape). Most working jobs can be stored on disk and the rest archived on tape for long-term storage. But the team nevertheless needs to pull back data from tape to disk on a regular basis.

David Foster, deputy head of IT at CERN, inside Tier Zero.

CERN primarily uses NAS storage over SAN, being that the latter has historically always been based on expensive, specialised infrastructure such as fibre channel rather than standard ethernet.

CERN also operates its own mass storage system called CASTOR, which currently stores some 500 million files, taking in two petabytes of raw data per month, or about 24 petabytes a year.

As the half-empty racks in this photo clearly demonstrate, footprint isn't the limiting factor at the CERN data centre, but power.

The extreme amount of power required for LHC experiments leaves only 3.4 Mw for the data centre. The data centre has been at full capacity for some time - but help is on the way thanks to 2.5Mw of additional power being made available via a new data centre in Budapest, Hungary.

The CERN Tier Zero facility includes a 24x7 operations centre - from where the IT team plans to remotely manage all operations at the new Wigner (Budapest, Hungary) data centre - the first room of which comes online before the end of the year.

All infrastructure at Wigner will be virtualised and managed from the Geneva facility.

The CERN Tier Zero data centre also showcases innovations from the organisation's 'OpenLab' industry partnerships. Currently HP, Intel, Oracle, Siemens and Huawei are engaged in early adoption programs with CERN.

Linked Articles

CERN to double capacity with Budapest data centre

Tags:

Victoria's first government tech chief steps down

WA Police Force to spend $30.8m on IT 'optimisation'

NSW Police to embark on $126m IT overhaul

Partner Content

Promoted Content AI in cybersecurity: weapon or shield?

Partner Content Australian organisations must act on security – or risk AI ambitions falling flat

Partner Content ElasticON Sydney 2025: Deriving value from your data with Search AI

AI Copilot: Breaking Down Silos & Securing the Future

Microsoft Copilot Partner Hub

Unlock SMB Success with Microsoft Copilot

Dave Stevens on Brennan's evolution and the need for Aussie tech unity

Sydney's ITKnocks on contact centre AI and the slow death of the IVR

"It's an exciting time to be part of the health and aged care sector"

Insicon founder Matt Miller on the coming 'tsunami' of compliance and educating boards about cyber security

Orro claims Australia first with managed digital asset discovery service

Latest Comments

Events

Tech in Gov 2025
Forrester's Technology & Innovation Summit APAC 2025
Security Exhibition & Conference 2025
Integrate Expo 2025
Digital As Usual Cybersecurity Roadshow: Brisbane edition

Most popular tech stories

Optus' first AI chief Samantha Lawson exits

NAB brings Pete Steel into new digital, data and AI exec role

ANZ's Maile Carnegie to exit next month

Adelaide University stands up its digital stack

Digital Nation announces finalists for its inaugural Digital Nation Awards
Kaaren Lewis and Mo Kandeel join Ingram Micro leadership team

Evergreen hungry for more Australian MSPs

Tech Data's Robbie Upcroft promoted as Andy Berry retires

Darktrace promotes Abhi Abhyankar to head of partnerships, alliances and channel for APJ

TAFE NSW, NESA land tech funding in state budget
Blackberry celebrates "giant step forward"

Laing O'Rourke brings IoT to hard hats

IoT Awards: How PGH Bricks and Pavers automated brick colour picking

Axis Communications opens experience centre in Sydney tech hub

Telstra to host IoT State of the Nation event on November 14

OTHER TECH SITES: techpartner.news | Digital Nation | IoT Hub

All rights reserved. This material may not be published, broadcast, rewritten or redistributed in any form without prior authorisation.
Your use of this website constitutes acceptance of nextmedia's Privacy Policy and Terms & Conditions.