Power outage downs OVH services

A major outage struck the world's third largest hosting firm OVH overnight, taking down services throughout Europe for about four hours.

Source: http://currentlydown.com/ovh.net

OVH hosts 260,000 servers across 22 data centres in 19 countries, and claims to support 155 out of the 1000 largest European companies.

The outage even took down the company's corporate website, though this was back up at time of writing.

OVH first informed its users on Twitter at 6.33pm AEDT on Thursday 9 November about an "incident" that was having an ongoing impact on its network.

It later indicated on its status page that an electrical issue at its Strasbourg site was behind the outage.

OVH chief executive Octave Klaba told customers that two separated 20kV lines were down, and said the firm was trying to restart generators for its Central Europe SBG1 and SG4 data centres.

The central Strasbourg data centre boasts 140,000 servers. 

The outage appeared to last for around four hours. It was largely centred on Europe, where OVH has data centres across several French cities, including Strasbourg, Paris, Gravelines and Roubaix.

Hello, incidents were detected in Roubaix and Strasbourg this morning. The normal functioning of certain services is affected. Octave Klaba has communicated from incident detection, via tweets and continues to do so. All our teams are on the bridge.

— Support OVH EN (@ovh_support_en) November 9, 2017

OVH launched in Australia in 2016 with its first footprint in one of Equinix's Sydney data centres, and in 2017 established a Melbourne office.

Update 11:30am AEST: OVH has revealed "two separate incidents" were to blame for the outage.

"The first incident impacted our Strasbourg site (SBG) and the 2nd Roubaix (RBX). In SBG we have 3 datacentres in operation and 1 under construction. In RBX, we have 7 datacentres in operation," it said.

The second problem was attributed to a software fault on third-party optical equipment from an unnamd supplier.

"We had a problem on the optical network that allows RBX to be connected with the interconnection points we have in Paris, Frankfurt, Amsterdam, London, Brussels. The origin of the problem is a software bug on the optical equipment, which caused the configuration to be lost and the connection to be cut from our site in RBX," OVH wrote.

"We handed over the backup of the software configuration as soon as we diagnosed the source of the problem and the DC can be reached again. The incident on RBX is fixed. With the manufacturer, we are looking for the origin of the software bug and also looking to avoid this kind of critical incident.

"We are in the process of retrieving the details to provide you with information on the SBG recovery time for all services/customers.

"We are sincerely sorry."