It seems as if every summer something seemingly innocuous happens in a Microsoft datacenter halfway around the world and it spreads through the service like wildfire, taking down access for vast numbers of customers. It happened at the end of June last year in 2014, where Exchange Online and Lync Online were down for hours, and it has just happened again this week.
The Outage
Here is how it started: at 5:28 PM Eastern time, Microsoft officially establishes an ongoing incident report:
Current Status: Engineers are investigating an issue in which some customers may be experiencing problems accessing or using Exchange Online services or features. This event is actively being investigated. More information will be provided shortly.
29 minutes later, at 5:57 PM Eastern, the company acknowledges that this outage is a reasonably widespread issue:
User Experience: Affected users are unable to connect to the Exchange Online service when using multiple protocols including Outlook, Outlook Web App (OWA), Exchange ActiveSync (EAS), and Exchange Web Services (EWS).
Customer Impact: A higher than average number of customers are reporting this issue. Analysis indicates that customers will likely have some users experiencing this issue.
33 minutes later, more explanation is given at 6:30 PM Eastern (keep in mind access has been down or very intermittent for a lot of users for over an hour at this point):
The investigation determined that a portion of infrastructure which facilitates authentication to the service is experiencing higher-than-normal resource usage. Engineers are analyzing service telemetry to determine what is causing the high resource usage.
30 minutes later, we get an admission that whoops! All of us are beta testers for Microsoft, didn’t you know that, and that a programmed update has knocked out e-mail for millions of people for an hour and a half. At 7 PM Eastern (92 minutes into this outage):
Engineers have determined that this issue may be related to a recent update to the service and are currently working to revert the update.
Another 37 minutes go by and at 7:37 Eastern time, two hours and nine minutes into the outage, we finally get some progress on fixing the issue:
Engineers are making progress in reverting the update which is believed to be causing this issue. The update has been reverted in the Latin American region and users hosted from that region should begin to see relief. Reversion of the update in the North American region is underway; as it progresses users there should also experience service restoration.
By 8:35 PM Eastern time, 58 minutes later and three hours and seven minutes, the fix is complete and Microsoft expected everyone to see usable e-mail again.
Engineers have reverted the update in all regions and affected customers should be experiencing service restoration. Service teams are validating that the configuration change has been applied correctly and that service health is recovering as expected.
Of course, it takes another one hour and four minutes for the mail queues that have built up during this outage to work their way through the filter and get delivered so now four hours and 11 minutes into the outage, it looks like things are turning up:
Engineers have validated that the configuration change has been applied to the affected infrastructure and are continuing to monitor service health. Affected customers should experience service restoration as mail queues continue to drain.
And finally, at 10:17 PM, essentially five hours later, Microsoft says the service is restored and healthy:
After validating that the configuration change was successfully applied to the affected infrastructure and mail queues have drained, engineers confirmed that service is restored.
Five Hours of Radio Silence for You
Did your users appreciate not being able to send or receive e-mail from within Outlook? Were they complaining to you? Did you have any idea what was going on? Was your first instinct to check the Office 365 portal for more information, or did you spin your wheels and waste time troubleshooting non-existent issues on your network before finally concluding you were up, it was Microsoft that was down?
During last year’s outage, one of the main complaints was that Microsoft was slow in updating the Service Health Dashboard about the issue and also providing regular updates about the progress to resolution and service recovery. We can say that during this outage, updates were provided—it looks like no more than an hour or so transpired between updates. But an hour is a long time, especially in a multihour extended outage like this one, and the Service Health Dashboard is not obvious in terms of discoverability and findability.
Gaining Visibility into the Issue
When you have hundreds of users shouting that their e-mail isn’t working, and you’re just the guy in the middle looking for information to pass on to your users, then you need all the context and data you can get. That’s where Mailscape365 from ENow Software comes in. The premise is simple but powerful: When you can notify your users early on about an incident in progress, and when your own reporting helps you keep those users up to date on progress toward restoration of service, you have created an environment that is a cut above the rest. Customers using Mailscape365 were instantly in the know through the product’s easy to use dashboard with green, yellow, and red indicators. No guessing, no trying to decipher Service Health Dashboard messages.
While an outage is never pleasant and never welcome, operating in the dark is worse. Don’t let that happen to you—as I mentioned, this certainly isn’t the first Exchange Online outage and you know it won’t be the last.
