Salesforce cloud outage caused by security change

A widespread Salesforce outage last week was caused by the rollout of a new permissions policy as part of defence-in-depth efforts that mistakenly locked users out of their accounts.

The four-hour outage on September 21, took out a number of Salesforce 'Clouds', along with Tableau and MuleSoft.

In a root cause analysis, published late last week, Salesforce revealed the outage was caused by an access permissions change.

"The change had unintended consequences," Salesforce wrote.

"While it was designed to add defence-in-depth, it inadvertently blocked access to other legitimate and necessary resources beyond its intended scope.

"The end result was a breakdown in communication between our services due to a lack of access permissions, causing failures within our systems.

"This restricted some of our customers from logging in and using the services."

The company explained that its "standard operating procedure mandates ongoing reviews and updates to security controls.

The change, the company admitted in its analysis, “was not identified in validation testing”, because whatever it changed couldn’t be pre-tested: “the standard change deployment pipeline could not be used to deploy this change.”

That meant a particular characteristic of the change wasn’t caught: that it was “applied at a level that impacted multiple systems”. 

Had the change fitted within Salesforce’s “automated deployment process … the change would only have affected a small number of services”.

Salesforce said this will be addressed with a new deployment and testing pipeline later this year.