On Friday, May 17th, Salesforce experienced system-wide degradation issues, impacting many Salesforce and Pardot users around the world.
What Happened
The Salesforce Technology team deployed an application database script that was supposed to be applied to a subset of Pardot users. It was inadvertently applied to all users in orgs that had Pardot provisioned at any time, either currently or historically. This deployment allowed for all users (including community users) to view and modify all Salesforce records.
Fearing that this would lead to unauthorized access, Salesforce made the decision to block access to the Salesforce service for all orgs (within both production and sandbox environments). Once the Technology Team was able to isolate the affected organizations, they restored access for non-affected customer organizations.
The Fix
On Saturday, May 18th, customers who were unaffected by the scripting error had their access restored in full. Affected organizations had access restored to their System Administrator users, but other profile types were still altered and did not allow for full access or functionality. Providing access to administrators allowed organizations to restore profiles and permissions internally, but in parallel, the company also began to execute automated provisioning to restore previous settings on a rolling basis.
As of Monday, May 20th, a subset of customers was still impacted, despite the automated provisioning. Many of these users ultimately had to choose between opening a support ticket (and waiting yet again) or restoring settings internally.
The Takeaway
Salesforce is usually a highly reliable service, with few outages in its history. This incident, in particular, caught Salesforce customers off-guard. Many users hoped to use their recently refreshed Sandboxes as a backup source, deploying permission and profiles back to production. However, the application database script impacted both production and sandbox orgs, leaving that option as a non-starter for most.
This entire incident has highlighted the importance of keeping multiple back-up options in place. Waiting on the Salesforce Technology team for can be crippling when an incident is widespread, as they have to deal with thousands of accounts.
While most users are back up and running in full capacity, it is still possible that you will run into issues with permissions and profiles. If you experience an issue, either open a ticket with Salesforce or contact us to let us know.
