NEWS
GOVERNMENT
SECURITY
AI
DIGITAL NATION
REPORTS
RESOURCES
PODCAST
EVENTS
ADVERTISE

BUSINESS CLOUD DATA CENTRE

EDUCATION FINANCIAL SERVICES HARDWARE

HEALTHCARE INDUSTRIAL NETWORKING

PROJECTS SOFTWARE STORAGE

STRATEGY TECHNOLOGY TELCO/ISP

HR MARKETING FINANCE LEADERSHIP GOVERNANCE EMERGING TECH

State of Security Cloud Covered

State of Security 2026 STATE OF SECURITY 2025 State of HR Tech

State of AI 2024 Special Reports Focal Points

MEDIA HUB PARTNER CONTENT PARTNER HUBS

RESEARCH Tech Partner Views iTnews TV

Security Breakfast Data & AI Breakfast

NEWS

BUSINESS CLOUD DATA CENTRE EDUCATION FINANCIAL SERVICES HARDWARE HEALTHCARE INDUSTRIAL NETWORKING PROJECTS SOFTWARE STORAGE STRATEGY TECHNOLOGY TELCO/ISP

GOVERNMENT SECURITY AI DIGITAL NATION

HR MARKETING FINANCE LEADERSHIP GOVERNANCE EMERGING TECH

REPORTS

State of Security Cloud Covered

RESOURCES

State of Security 2026 STATE OF SECURITY 2025 State of HR Tech State of AI 2024 Special Reports Focal Points MEDIA HUB PARTNER CONTENT PARTNER HUBS RESEARCH Tech Partner Views iTnews TV

PODCAST EVENTS

Security Breakfast Data & AI Breakfast

ADVERTISE

Log In

Email:

Password:

Remember me | Forgot password?

Don't have an account? Register now!

Home
Galleries
Technology
Hardware

Photos: The data centre behind CERN's LHC

Inside Tier Zero.

on Aug 3 2012 7:00AM

CERN's 'Tier Zero' data centre is located on the same site as the Large Hadron Collider, a high-energy particle accelerator set in a 27km tunnel between Switzerland and France, near Geneva. The €7.5 billion project aims to prove or expand upon several key theories in the field of particle physics.

Six detectors - each operating discrete experiments - are housed within the LHC. Each operates its own server farm on site, for real-time filtering of results from particle beam collisions occurring every 25 nanoseconds. From there, results that appear to be of interest are sent in close to real-time to the CERN data centre.

The CERN data centre, or 'Tier Zero', was built in the 1970s. Its high ceilings once accommodated mainframes and a conveyer belt delivering rolls of magnetic tape between two floors.

Today, it forms the central hub of the Worldwide Large Hadron Collider Computing Grid, which harnesses the power of some 160+ computing facilities around the globe.

The lobby of the data centre includes a Google Earth map of jobs sent from CERN's Geneva data centre (Tier Zero) to about a dozen Tier One data centres (for near real-time processing of data) and over 150 Tier Two data centres (for batch analysis).

Some 1.5 million jobs are sent per day. Even when the LHC isn’t in operation, analysts are requesting access to older data to test new theories.

The core of Tier Zero's contribution to the Worldwide Large Hadron Collider Computing Grid is rows and rows of disk and CPUs. It is built almost entirely out of commodity computing products rather than highly sophisticated supercomputers. The middleware not only distributes jobs but also records what experiments have used what compute resources for the purposes of chargeback.

The Worldwide Large Hadron Collider Computing Grid takes up most of the raised-floor style CERN data centre. A lot of plant (generators, telecoms equipment) is housed externally, with UPS stored in a basement room.

The CERN data centre houses more than 60,000 processing cores and roughly 60,000 disks.

The facility also contains about 600 10Gbps connections.

Every system (server) is connected at 1Gbps, and each is uploaded to a switch with an uplink at 10Gbps to a massive routing core (a farm of interconnect routers), which boasts 7-8 Tbps of switching capability.

CERN was one of the very first internet sites in Europe. CERN staff have been using the Internet since the early 80s when it was primarily an academic research network.

As such, Tier Zero became one of the first digital exchanges - this is where many ISPs put their equipment to connect with other ISPs. This head start had a snowball effect and Tier Zero remains a popular ISP exchange.

CERN is currently upgrading its core routing infrastructure to support synchronous connectivity with a new facility being built in Budapest, Hungary. Pictured here is part of the network backbone for the grid and the accelerator itself.

The lower floor was originally a tape library connected to the second floor via a conveyer belt. Operators on the upper floor would send down a call for a specific tape, operators in the lower floor would collect it from the racks and put it on the conveyer belt.

Today, this lower floor is primarily used for disk and tape storage.

CERN uses tape storage for long term archiving. It stores some 60 petabytes on tape at the CERN site.

The tape robot pictured here takes requests from servers upstairs, scoots up and down the tape machine scanning the bar codes attached to the spine of the tape cartridges and mounts the appropriate cartridge.

CERN currently uses three manufacturers for its tape arrays – IBM, StorageTek and SpectraLogic. "We don't believe in having one supplier for anything," said David Foster, deputy head of IT at CERN.

The lower floor also includes a Linux compute cloud for use by CERN staff and those of its collaborating partners. CERN employs 2250 staff, but the CERN IT team serves some 8000 users, including students and staff working on-site at LHC experiments.

As the price of disk has dropped over time, CERN now runs at least 20 petabytes of disk on-site at Tier Zero (vs 60 petabytes of tape). Most working jobs can be stored on disk and the rest archived on tape for long-term storage. But the team nevertheless needs to pull back data from tape to disk on a regular basis.

David Foster, deputy head of IT at CERN, inside Tier Zero.

CERN primarily uses NAS storage over SAN, being that the latter has historically always been based on expensive, specialised infrastructure such as fibre channel rather than standard ethernet.

CERN also operates its own mass storage system called CASTOR, which currently stores some 500 million files, taking in two petabytes of raw data per month, or about 24 petabytes a year.

As the half-empty racks in this photo clearly demonstrate, footprint isn't the limiting factor at the CERN data centre, but power.

The extreme amount of power required for LHC experiments leaves only 3.4 Mw for the data centre. The data centre has been at full capacity for some time - but help is on the way thanks to 2.5Mw of additional power being made available via a new data centre in Budapest, Hungary.

The CERN Tier Zero facility includes a 24x7 operations centre - from where the IT team plans to remotely manage all operations at the new Wigner (Budapest, Hungary) data centre - the first room of which comes online before the end of the year.

All infrastructure at Wigner will be virtualised and managed from the Geneva facility.

The CERN Tier Zero data centre also showcases innovations from the organisation's 'OpenLab' industry partnerships. Currently HP, Intel, Oracle, Siemens and Huawei are engaged in early adoption programs with CERN.

As the half-empty racks in this photo clearly demonstrate, footprint isn't the limiting factor at the CERN data centre, but power.

Linked Articles

CERN to double capacity with Budapest data centre

Tags:

Kmart Group to expand RFID tagging to more products and to Target

Microsoft teases new era of AI-driven devices

PsiQuantum to build computer at Moreton Bay

Partner Content

Partner Content AI is delivering business value today

Promoted Content Why resilient communications are becoming critical infrastructure for modern enterprise IT

Partner Content Agile isn’t the problem: why projects still fail, and what’s missing

Promoted Content From test case to control tower: How DXC and ServiceNow are governing enterprise AI at scale

Partner Content Thomas Peer Solutions unveils data cloud platform and executive leadership forum for 2026

Promoted Content You meet the security standard. Shame no one can see it

Sydney-based AI-cloud waste startup raises $3m

Brennan uses NiCE to modernise its contact centre

Impact Awards: Tecala slashes customer response times for fintech IQumulate

Interactive introduces private cloud platform

Digital61 expands cybersecurity portfolio

Latest Comments

Events

iTnews State of Security Breakfast
iTnews State of Data & AI Breakfast
Forrester's AI Forum Sydney
The 2026 iAwards
Integrate 2026

Most popular tech stories

Treasury Wine Estates to go big on digital, data and AI

ABC drops Salesforce for Braze

Chemist Warehouse's AI tool for HR becoming a "standard pattern"

Virgin Australia, Wesfarmers strike OpenAI agreements

Stockland builds AI assistant as a bridge into SAP
HamiltonJet partners with digital services provider Fortude

SentinelOne signs distribution agreement with Sektor

Rapid7’s new SIEM combines exposure management with threat detection

The techpartner.news podcast, episode 3: Why security consultancy founder Kat McCrabb started with the hard stuff

Bluechip Infotech enters final stage of Goodson Imports acquisition
Blackberry celebrates "giant step forward"

'Touch-free' smartphone controlled with head movements

Photos: Australian industry explores data for net zero

Telstra Purple acquires IoT specialists Alliance Automation, Aqura Technologies

Govt launches consumer tech label program for smart devices

OTHER TECH SITES: techpartner.news | Digital Nation | IoT Hub

All rights reserved. This material may not be published, broadcast, rewritten or redistributed in any form without prior authorisation.
Your use of this website constitutes acceptance of nextmedia's Privacy Policy and Terms & Conditions.