Amazon forced to reboot EC2 to patch Xen bug

Amazon Web Services has started urgently rebooting Elastic Compute Cloud (EC2) instances in all its regions, after an unknown bug was discovered in the open-source Xen virtualisation platform used by the company.

The cloud giant emailed EC2 customers today to warn about the reboot, which it called "required host maintenance".

"One or more of your Amazon EC2 instances are scheduled to be rebooted for required host maintenance," AWS advised in the email.

"The maintenance will occur sometime during the window provided for each instance. Each instance will experience a clean reboot and will be unavailable while the updates are applied to the underlying host."

It said each reboot should take only a few minutes and instances would then return to normal operation, with all configuration and data retained.

"We will need to do this maintenance update in the window provided. You will not be able to stop/start or re-launch instances in order to avoid this maintenance update."

However, an AWS customer who declined to be named said the reboot was not simply maintenance and was instead due to an unspecified bug in the Xen platform, XSA-108, which will not be revealed until October 1 United States time.

Update 29/6:  AWS evangelist Jeff Barr confirmed on the company’s blog that the reboots are due to unspecified problems with Xen.

Barr said "these updates must be completed by Oct. 1, before the industry notice comes out on Xen update XSA-108. The issue in that notice affects many Xen environments, and is not an AWS-specific issue. Following security best practices, the details of this update are embargoed until then."

The blog post was taken down shortly after publication.

In a separate security bulletin, AWS advised that it has patched the Bash command shell in its AMI Linux distribution against the Shellshock vulnerability.

Cloud management company Rightscale confirmed AWS was rebooting a substantial number of EC2 instances on its blog.

Rightscale engineer Thorsten von Eicken said the reboots would take place over the next few days, starting September 27 Australian time and ending on October 1.

"As usual, AWS is totally tight-lipped about the underlying cause. It seems obvious that the company is patching a security vulnerability, but it will not disclose which one until October 1 — that is, after they have patched all hosts," von Eicken said.

Von Eicken noted that not all AWS instance types  - such as T1, T2, M2, R3 and HS1 - would need to be rebooted.

However, all AWS regions and availability zones are affected, von Eicken said.

iTnews has contacted AWS for comment.