Wrongly removed key caused Azure Active Directory outage

A wrongly removed digital key for Azure Active Directory was the reason why Microsoft customers were locked out of their applications on Tuesday this week.

Azure AD keys for cryptographic signing operations not in use are removed automatically for security reasons.

However, one such key was used in a complex cloud-to-cloud migration and marked as "retain" for a longer period than usual.

A bug in the Azure AD ignored the "retain" state, and the key was removed, which meant users were no longer able to authenticate and use their applications.

While Microsoft rolled back the key metadata just over two hours after the problem was identified, cached metadata led to residual impact for a further twelve hours.

An update that invalidated the cached key metadata entries, together with a forced refresh, cleared the residual impact for customers.

"We understand how incredibly impactful and unacceptable this incident is and apologise deeply," Microsoft said.

An incident that flowed on from the Azure AD incident saw OneDrive for Business, Sharepoint and Teams customers' files being removed from their shared folders, and placed in Recycle Bins.

Microsoft has resolved the issue, which was caused by file synchronisation to local machines not working as expected.

Company support staff said that either restarting customers' machines, or waiting 24 hours for an automatic re-sync will fix the problem.

The company is planning to apply additional protections to the Azure Safe Deployment Process system to prevent a repeat of this week's outage.

Better communication with customers on fault investigation and remediation progress, and making sure messages are consistent across Azure, Microsoft 365 and Dynamics 365 is also promised.

This week's outage follows major ones for Azure Active Directory, with flow-on effects to 365 services, in June, September and October last year.