Microsoft 365 Online Apps & Teams Admin Center Access Issues
On April 20, 2023, at approximately 7:56 AM ET, Microsoft tweeted via their account @MSFT365status...
On January 13, 2023, at ~7:12 AM ET, Microsoft communicated via tweet (@MSFT365status) that they were investigating an issue in which some users were unable to "access" application shortcuts via the Windows Start menu and/or the Taskbar.
first
We're investigating an issue where users are unable to access application shortcuts on the Start menu and Taskbar in Windows. For more details and updates, please follow the SI MO497128 in your admin center.
— Microsoft 365 Status (@MSFT365Status) January 13, 2023
Twitter and Reddit chatter from the public was immediate, given the large number of users impacted. Many responses on Twitter were from system admins and IT teams suggesting the matter was a bit more complicated than Microsoft first disclosed, and the general community sentiment on social media was that this would be a very busy Friday for most organizations' help desk teams.
Approximately one hour after their first message, @MSFT365status provided a second message to the community which stated that a "specific rule" was the cause of the service incident, that a rule reversion was in place, and that Microsoft was continue to investigate matter. But no expanded explanation was provided on Twitter by Microsoft initially. Additional information on the service incident could be found in the Microsoft admin center, provided you had or your team had access to the admin center.
We've identified that a specific rule was resulting in impact. We've reverted the rule to prevent further impact whilst we investigate further. For more information, please follow the SI MO497128 in your admin center.
— Microsoft 365 Status (@MSFT365Status) January 13, 2023
As this issue persisted, more information came to light. The "specific rule" was in fact the Microsoft Defender for Endpoint ASR rule which appeared to have triggered in error and affected app shortcuts. This triggered Defender ASR rule was deleting application shortcuts from the desktop, from the Start menu and from the task bar.
Microsoft's next message indicated that the reversion was still in progress and that it would take several hours for the reversion to complete, but they gave no estimate as to when the matter would fully resolve.
SI MO497128The revert is in progress and may take several hours to complete. We recommend placing the offending ASR rule into Audit Mode to prevent further impact until the deployment has completed. For more details and instructions, please follow the SI MO497128 in your admin center.
— Microsoft 365 Status (@MSFT365Status) January 13, 2023
Approximately 5 hours after their first report, and with still no issue resolution, Microsoft repeated that the rule reversion was still in progress, and that any impacted customers should place the "offending" ASR rule into audit mode for the time being. Social media responses from a very frustrated community continued to pour in.
We're reviewing options to expedite the deployment of the change which contains the reverted rule. We still recommend that you take action to place the offending ASR rule into Audit Mode. Please follow the SI MO497128 for more details and instructions.
— Microsoft 365 Status (@MSFT365Status) January 13, 2023
It was not until approximately 10 hours after first report that Microsoft was able to communicate to the public that their "fix" had been completed and that no additional impact should occur. Responses from many on social media were quick to respond that no remediation effort on Microsoft's behalf could repair the damage already done.
Oddly enough, Microsoft was still unable to provide additional and specific details as to the root cause and they indicated that their investigation continued.
The fix has completed its deployment, which will prevent additional impact from occurring. We're also continuing our investigation into the issue, and further detail can be found under MO497128 in the Microsoft 365 admin center.
— Microsoft 365 Status (@MSFT365Status) January 13, 2023
After several status quo tweets by Microsoft, they did post two meaningful updates on January 14 and 15, in which they provided customers links and information on how to recreate start menu links and perform other steps towards returning to business as usual. This was progress of a sort, despite the continued social media responses from a beleaguered community, especially those IT professionals working through the weekend on the issue.
We've confirmed steps to recreate start menu links for a significant sub-set of the affected applications that were deleted. Please visit https://t.co/FYLP1Jvg7Y for further guidance. Further updates can be found in the admin center under MO497128.
— Microsoft 365 Status (@MSFT365Status) January 14, 2023
We've published an update to https://t.co/Vn2Jqq71r4 to include additional details and steps to deploy the script using Microsoft Intune. Please continue to visit the site for updated guidance as well as MO497128 in the Microsoft 365 admin center.
— Microsoft 365 Status (@MSFT365Status) January 15, 2023
Some two days after their first reporting of the issue, and with an entire weekend of overused help desks across many organizations worldwide, Microsoft messaged that a final version of a script was available to aid customers to hopefully recover any affected shortcut files.
We've finalized version 1.1 of the script intended to aid customers in recovering affected shortcut files and have updated https://t.co/FYLP1Jvg7Y with the script details. We recommend reviewing https://t.co/FYLP1Jvg7Y and MO497128 in the Microsoft 365 admin center for more info.
— Microsoft 365 Status (@MSFT365Status) January 16, 2023
On January 17, 4 days after the service incident began, Microsoft communicated their last message as to MO497128, indicating that yet another version of the remedy script was available.
We've finished version 2.0 of the script with expanded functionality and have updated https://t.co/Vn2Jqq71r4 with the new version's details. We recommend reviewing https://t.co/Vn2Jqq71r4 and MO497128 in the Microsoft 365 admin center for more info.
— Microsoft 365 Status (@MSFT365Status) January 17, 2023
There have been no further communications from @MSFT365status as to this incident.
In a cloud-world, outages are bound to happen. While Microsoft is responsible for restoring service during outages, IT needs to take ownership of their environment and user experience. It is crucial to have greater visibility into business impacts during a service outage the moment it happens.
ENow’s Microsoft 365 Monitoring and Reporting solution enables IT Pros to pinpoint the exact services effected and root cause of the issues an organization is experiencing during a service outage by providing:
Identify the scope of Microsoft 365 service outage impacts and restore workplace productivity with ENow’s Microsoft 365 Monitoring and Reporting solution. Access your free 14-day trial today!
On April 20, 2023, at approximately 7:56 AM ET, Microsoft tweeted via their account @MSFT365status...
On July 20, 2022, at ~9:47 PM EST, Microsoft communicated via tweet (@MSFT365status) that they were...