One-second pause takes out websites

Admins, Java apps unprepared.

An attempt to keep Earth's atomic clocks in tune with changes to the planet's rotational speed was behind outages to a number of websites and programming platforms over the weekend.

The International Earth Rotation and Reference System Service held clocks back by a single second on 11.59pm UTC on Saturday night, one of several such additions since the 1970s.

Though a regular occurence, Opera Browser engineer Marco Marangiu said many applications and systems continued to rely on the wrong assumption; that every minute lasts exactly for 60 seconds.

As a result, such systems can crash badly during a leap second insertion.

Popular social bookmarking site Reddit was one of many that experienced problems for a few hours due to the leap second.

We are having some Java/Cassandra issues related to the leap second at 5pm PST. We're working as quickly as we can to restore service.

— reddit status (@redditstatus) July 1, 2012

Other large sites such as Foursquare, Yelp, Meetup and StumbleUpon also experienced outages, together with Gawker Media’s websites, according to Buzzfeed.

The paused second appeared to be fatal for Java apps, with Mozilla Foundation site reliability engineer Eric Ziegenhorn noting that “Java is choking on leap second”.

“Servers running Java apps such as Hadoop and ElasticSearch and Java doesn’t [sic] appear to be working," Ziegenhorn wrote.

"We believe this is related to the leap second happening tonight because it happened at midnight GMT.”

Mozilla has pushed out a bug fix to sort out high CPU usage for MySQL and Java servers.

ServerFault also published workarounds, and reported Linux-based servers experiencing problems.

Sorting out the problem of adding leap seconds is a complicated endeavour however; Google said it would “smear” the leap second, gradually adding milliseconds over a day rather than a full second at once.