Happy New Year! With the recent change to the new year, a nasty bug similar to the Y2K bug (Y2K22?) appeared in Exchange 2016 and Exchange 2019. On 01/01/2022 at 0:00 hour UTC a date conversion failed inside the product, causing the anti-malware agent to fail. As a result, the Exchange transport service was no longer able to process email and mail started to queue on the Exchange server. This also happens on Exchange 2013, but email is not queued and therefore the issue is not directly visible.
Since Exchange works with UTC time under the hood, you might have seen this at a different time depending on your geographical location. When this happens, Event ID 5300 (source FIPFS) is written to the event log as shown in the following screenshot:
The initial workaround was to disable the anti-malware transport agent on the Exchange servers and restart the transport service and the queues are processed. While this works well, it poses a security risk. We all know inbound email is a well-known attack vector.
Dynamic discussions quickly appeared on Twitter, Reddit and LinkedIn of course and also on internal Microsoft Distribution lists. Please note that it was still January 1st. Microsoft was quickly aware of the issue and during the day a fix was made available for this. The fix was actually a PowerShell script that must be run on all Exchange servers. This script stops the transport service, deletes all files regarding anti-malware, downloads then updates anti-malware signature files and starts the transport service again. In a typical situation there’s no need to reboot the Exchange servers, but several admins reported they had to reboot (so had I).
One pitfall, when the anti-malware agent was disabled earlier, it must be re-enabled after running the PowerShell script. It does not automatically re-enable the anti-malware agent.
So how could this have happened? In my opinion it is a very simple bug that one should not expect from a professional developer. We don’t know how long ago this particular piece of code was written, maybe the original developer is long gone, and nobody was aware. On the other hand, this is what code reviews are for.
Another question is why mail flow halts on Exchange 2016 and Exchange 2019, and not on Exchange 2013. The bug also appeared on Exchange 2013, but the outcome was very different.
According to Microsoft this bug did not impact (or did not appear in) Exchange Online. There’s a possibility that it did appear in Exchange Online but was solved quickly so we haven’t noticed it. The code base in Exchange Online is also different, so this can contribute as well. Or maybe a different approach to anti-malware is used in Exchange Online.
The only thing we can do in a situation like this is rely on a good monitoring solution. When Exchange halts for whatever reason, make sure you are warned in time. Especially with queueing, where queues expire in 48 hours (which in this case is Sunday night!).
With email being one of the most mission-critical tools for organizations today, how do you ensure vital business communication stays up and running? How do you demonstrate to senior management that additional resources are needed to meet growing demand or that service levels are being met?
Developed by Exchange architects with direct product input from Exchange MVPs, ENow's Mailscape makes your job easier by putting everything you need into a single, concise OneLook dashboard, instead of forcing you to use fragmented and complicated tools for monitoring and reporting. Easy to deploy and intuitive to use, get started with Mailscape in minutes rather than days.
ACCESS YOUR FREE 14-DAY TRIAL and combine all key elements for your Exchange monitoring and reporting to keep your messaging infrastructure up and running like a pro!
PRODUCT HIGHLIGHTS