Top Support Issues for Exchange 2013/2016

This time next year we’ll (hopefully) be in the midst of project planning for the newly released Exchange Server 2019. After a scheduled late-2018 release, many organizations will be looking to learn, test, deploy, and administer the new on-premises Exchange version after returning from their holiday break. As someone who works in the Exchange Support and Consulting world, this (Jan 2017) is actually a relatively slow period for us. Exchange 2016 has been released for a couple years, its bugs and quirks mitigated, and the rush of early adopter migrations have passed; leaving only the steady flow of migration and deployment work driven by company financial scheduling rather than the Exchange product release cycle.

I’ve always enjoyed working in a support organization during a new product release, as I get to experience the hurdles encountered as customers navigate life on “The Bleeding Edge.” Exchange 2007 brought us a wave of customers dumbfounded with Exchange Management Shell, asking themselves why they were using Windows but still being asked to learn shell. In addition, newly added web services which required shell configuration and a trusted SSL connection meant Exchange Professionals now required extensive PKI and shell expertise. Exchange 2010’s release seemed to be dominated by two common types of support issues in my experience; load balancing and TMG configuration troubles. The questions of how to go about load balancing a CAS Array, OWA, Outlook Anywhere, EWS, ActiveSync, and SMTP seemed to trouble many Administrators and Consultants. Combine that with the growing desire to securely publish Exchange via a Reverse Proxy and you had some of my most commonly seen issues during Exchange 2010 migrations and deployments.

A common theme I’ve found from each of these eras is how these skills are not becoming obsolete. PowerShell skills are more important than ever, as is PKI knowledge. Exchange Server still requires proper load balancing, and while TMG is a dead product, load balancers have become popular reverse proxy/web-publishing solutions. Exchange 2010 was also the first product version to introduce a “Hybrid Configuration” which mandated an Exchange Professional add Cloud/Identity to their skillset. Of course, it too has posed its collection of support and deployment challenges. My point here is that as new products emerge, we’re better prepared for the technical challenges ahead by making sure we’ve learned from the past. With that said, I’d like to detail some of the common support challenges I’ve seen during the lifetime of Exchange 2013/2016. This experience comes not only as a Principal Engineer for a global support organization, but also as a moderator of an active Exchange community.

Before beginning, I’d like to defend the Exchange product itself for a moment. I feel Exchange Server is an extremely well made product, but it often gets a bad reputation as a beast to manage and deploy. I personally feel this isn’t due to it being a complicated piece of software (I would argue SQL Server has a steeper learning curve to master the product itself), but more because of the MANY different pieces of technology it interacts with. Consider the below list of components that make-up a typical medium-to-large Exchange environment:

• Internal/External network components
• Load Balancer, Reverse Proxy, Firewall, WAN Accelerator, VPN, etc.
• Storage Solutions
• DAS, SAN, Hyper Converged, etc.
• PKI
• Client Connectivity, Secure SMTP (TLS), etc.
• Windows Client and Server Operating Systems
• Various windows features such as IIS, Bitlocker, Failover Clustering, DNS, etc.
• Windows Active Directory/Azure Active Directory
• The health of Exchange lives and dies on the health of Active Directory
• Virtualization
• Hyper-V, VMware/ESX, Client VDI solutions, etc.
• Various email clients
• Outlook, Outlook for Mac, Outlook on the Web, OWA for iPhone/Android, Outlook for iPhone/Android, POP/IMAP, VDI environmental challenges, etc.
• Non-Exchange related mail flow technologies
• Anti-SPAM Solutions, SPF, PTR, DKIM, DMARC, etc.
• Various optional solutions which integrate with Exchange
• Email Archiving and Compliance, Outlook Add-ins, VoIP/Voicemail systems, various line-of-business applications, etc.

Whether these components are managed by different teams or one multi-talented individual, they still account for a fairly complex overall solution; which means there are many failure points outside the control of the Exchange Server Product Team. The majority of the Exchange escalations I’ve received in my career have either been related to a lack of product understanding, or an issue with one of the above external technologies.

Now for the list of my most commonly encountered support issues during the lifetime of Exchange 2013/2016; in no particular order:

Note: I’m combining the 2013 and 2016 versions of the product because many experts would agree they’re extremely similar.

.NET

I’ve never been much of a programmer. Outside of college, I’ve never written a single line of code. Before Exchange 2013 was re-written into Managed Code, I had only viewed .NET as something that was typically a prerequisite to another piece of software I needed to install. I still know little of the programming language itself but what I do know is the health of Exchange 2013, Exchange 2016, and the future Exchange 2019 relies on the health of .NET Framework. However, as .NET is managed by a separate product team at Microsoft than Exchange Server, and also distributed via Windows Updates, it has become very problematic for customers to ensure they’re on a version of .NET that’s supported for Exchange Server.

Aside from being unsupported by Microsoft, system stability issues can occur when running a version of .NET which is not supported on your currently installed Cumulative Update. In my experience, it really depends which version we’re talking about, as I’ve seen a particular CU work fine with a particular unsupported .NET version; yet in other scenarios I’ve seen a system perform extremely slow and experience Exchange process crashes when running an unsupported .NET version. Though it does seem recent .NET releases haven’t caused any issues in my personal experience. However, it’s best not to risk unexpected issues and to keep your system in a supported state.

You can find which versions of .NET are supported with which versions of Exchange Server on both the Exchange Server System Requirements TechNet article and the Exchange Server Supportability Matrix. However, determining the installed version of .NET isn’t as simple as looking at “Add/Remove Programs.” There are a few ways to see which version is currently installed. This blog post of mine provides a one-liner PowerShell command to detect the installed version; or scripted methods exist if you prefer. Should you find yourself needing to update in a supported manner, this post from MVP Michel de Rooij details the supported upgrade paths for Exchange CUs and .NET.

At Microsoft Ignite, Microsoft alluded to these issues and expressed understanding for customers’ pain while committing to work more closely with the .NET team to ensure the process is easier in the future. Therefore, while we may see changes to this topic in the coming months, it’s still important to understand these existing challenges.

Virtualization Mishaps:

As a Support Engineer who has written multiple articles about Exchange Virtualization design and performance, it should come as no surprise it ranks near the top of my list. This is not to bash virtualization at all, it’s an amazing technology with numerous benefits. But much like a Formula 1 racer, in inexperienced hands it can be dangerous. No need to rehash my aforementioned posts but the common types of issues haven’t changed much. Common issues include:

• Resource contention (overcommitting CPU and RAM)
• EX: Experiencing issues after moving a properly-sized Exchange VM to a different (overloaded) host
• Suggest using proper monitoring and affinity rules/policies
• Storage misconfiguration
• Follow vendor storage network guidance (jumbo frames, queue depth, etc.)
• Unsupported technologies
• Avoid Checkpoints/Snapshots on Exchange VMs

As these issues aren’t unique to Exchange, my only remediation suggestion would be for IT leaders to invest in their people. Hiring IT Professionals skilled in virtualization and investing in continuing education is always a winning combination. I’ve personally found there to be a huge difference between someone who can be an administrative user of virtualization (create a VM, add/remove resources, etc.) and someone who truly understands the technology. One of my favorite IT-related articles, “Rise of the Expert Beginner” really highlights this skill gap.

Patch and Upgrade:

Patching and updating Exchange Server has become a point of frustration to many Exchange Professionals. The process is made challenging by the following factors:

• New Servicing and Cumulative Update Cadence
• Not every company can update Exchange often enough to remain within the N-1 rule
• Hybrid configuration complicates matters as being updated is required for full cloud compatibility
• Confusion around what is supportability, serviceability, etc.
• Older Cumulative Update files are not made available to download as security updates are not provided for them
• File download availability follows an N-2 rule
• Conflicting Anti-Virus software
• Proper exclusions should be in place
• It’s recommended that any Anti-Virus active scanning be disabled while an Exchange update is being performed

While some of these are operational challenges, they’re common support call drivers as well. Crashed updates due to Anti-Virus killing the process, Server Component States not properly re-activating after an update, or even failures resulting from other infrastructure upgrades.

Support Issues on the Decline:

Fortunately, advances in the Exchange Server product have led to decreases in the frequency of certain common call generators. Two of the most frustrating Exchange support issues of the past 15 years have been those dealing with Public Folders or Database Recoveries. Public Folder issues have decreased because their most complicated aspect (replication) has been done away with in Modern Public Folders (Exchange 2013/2016). There are no more Public Folder replication storms or complex recovery scenarios to navigate as Public Folders are now specialized mailboxes rather than objects replicated between Public Folder Databases (also removed from Exchange 2013/2016). Modern Public Folders, are in my opinion, relatively easy to configure and administer. The only challenges come in the largest of Public Folder environments where performance, scale, and distribution become concerns. When encountering Modern Public Folder support issues, I recommend the following two Microsoft articles:

• Modern public folder deployment best practices
• Modern public folders logging and when to use it

As for database recovery issues, those have also declined due to “under the hood” architectural changes and improvements in the product. The Information Store services are now so robust that a loss of power or unexpected shutdown rarely result in a database that requires manual repair. Combine this with a Database Availability Group being deployed and an Exchange Professional may never again need ESEUTIL.

Summary:

While I have high hopes that Exchange Server 2019 will be an excellent and stable release, given the stability of Exchange 2016, we shouldn’t forget the issues of the past. Most of the support call drivers I mention above will still be relevant in Exchange Server 2019, and as the saying goes, “Those who ignore the past are doomed to repeat it.”

For a more thorough Exchange Troubleshooting reference, check-out the “Exchange Server Troubleshooting Companion” which I co-authored with fellow MVP Paul Cunningham.

Looking to get ultimate visibility into your Exchange environment? Get started with a free trial now: