Overcoming corruption during mailbox moves

The following blog post is a trimmed-down excerpt from the eBook "Exchange Server Troubleshooting Companion":

Mailbox corruption is not a new problem for Exchange Administrators, Support Engineers, and Consultants. Causes of corruption can vary and include:

• Hardware/Storage failures
• Power failures
• File System failures or corruption
• Faulty ActiveSync clients
• Faulty third-party Outlook add-ins
• Anti-Virus software (faulty or missing proper exclusions)

Much of this corruption may go undetected during normal operations. Event logging for physical database corruption can be found in the Application logs, but logging for corruption within individual mailboxes is somewhat sparse. In fact, most people will only discover mailbox corruption when attempting to move a mailbox either to new databases within the company or across Exchange organizations. They’ll find the mailbox moves will fail due to bad items being encountered, and because the default BadItemLimit will be zero for all move requests it’s not at all uncommon. In some cases, after a database corruption event where the mailbox database was left with bad checksums or an ESEUTIL /P had to be run against it, the mailboxes within the database are left in a corrupted state (hence Microsoft’s new support policy stating that a database must be vacated after an ESEUTIL /P was run against it). Let’s discuss behaviors when corruption is present as well as how to recover from them.

Mailbox Quarantine

Often times, it's possible mailboxes may become so corrupted and unstable that they become quarantined by Exchange. A quarantined mailbox means the user will be unable to access the mailbox using any client. A mailbox will become quarantined if it causes the Information Store processes to hang or crash repeatedly. That mailbox will then be inaccessible by any mail client (such as Outlook, Outlook Web App, ActiveSync, etc.) until a given time period has expired. The default for this “penalty box” time period is six hours and can be customized by modifying the registry. The Mailbox Quarantine feature was created in Exchange 2010’s lifetime but options were limited when it came to detecting & configuring it. You were limited to searching the below Windows Registry key for quarantined entries and then clearing them by deleting the registry entry for a particular mailbox.

HKLM\SYSTEM\CurrentControlSet\Services\MSExchangeIS\<ServerName>\Private-{dbguid}\QuarantinedMailboxes\{mailbox guid} 

Fortunately, with Exchange 2013 came cmdlets to help administer Mailbox Quarantine. These commands were:

• Enable-MailboxQuarantine
  
  
• Disable-MailboxQuarantine 

These commands replaced the manual deletion of registry keys as a means to disable mailbox quarantine for mailboxes. In addition to the above commands, the below command can be used to detect which mailboxes are currently quarantined.

Get-Mailbox | Get-MailboxStatistics | Select DisplayName, IsQuarantined | Format-Table –AutoSize

Types of Corruption 

Sometimes you’ll have mailboxes that frequently get quarantined, requiring they be repaired. Let’s first discuss the two categories of corruption in an Exchange mailbox or mailbox database - physical corruption and logical corruption. 

The analogy I like to use when discussing Exchange physical vs. logical corruption is that of a damaged book. If a book’s pages are torn out and its binding is damaged, I compare that to physical corruption in a database. This is where we would need to run ESEUTIL against the database to repair its physical structure, either by running an ESEUTIL /R or the dreaded ESEUTIL /P (we won’t be covering ESEUTIL in this post, as the focus is mailbox corruption not database corruption). Now, once the book’s pages and binding have been repaired (with all pages now in the correct order), it could still have logical corruption if the words on the pages don’t make sense to the reader, if the letters are smudged, the words are out of order, or it’s now in the wrong language. For this, Exchange had a utility called ISINTEG, which would correct any logical corruption that resulted in Exchange being unable to properly decipher and process data in the Jet database. The symptoms of logical corruption were often display- or search-related issues in Exchange clients, or in some cases, messages not displaying at all. I worked with one customer who were unable to sync ActiveSync devices because of a combination of physical and logical corruption in their mailboxes. While ISINTEG was certainly useful for recovering from this, starting with Exchange 2010 SP1, ISINTEG was replaced with the New-MailboxRepairRequest Exchange Management Shell command.

Repairing Logical Corruption in a Mailbox

New-MailboxRepairRequest allows you to target individual mailboxes instead of taking the entire database offline. However, it should be known that while an individual mailbox is being scanned, the mailbox data is made unavailable to the end user. The New-MailboxRepairRequest command takes the following format: 

New-MailboxRepairRequest -Mailbox <MailboxName> -CorruptionType <CorruptionType>

Available Corruption Types:

• AggregateCounts
• Searchfolder
• Folderview
• Provisionedfolder

Note: I typically include all corruption types using commas to separate them. 

In some situations, you can also use the CorruptionType of LockedMoveTarget if you encounter a mailbox move that has become locked.

In Exchange 2010, you monitored the progress of mailbox repair requests in Event Viewer. However, in Exchange 2013/2016, you instead use the Get-MailboxRepairRequest command. A common example of the command would be:

Get-MailboxDatabase | Get-MailboxRepairRequest | Format-Table Identity

Repairing Physical Corruption

Should a repair request not resolve the corruption issues, we would need to repair physical corruption by moving the mailbox to a new database. This action effectively creates a new mailbox by copying the old mailbox data to it. Take care however, as this action will likely result in mailbox data being purged if it is deemed corrupt. You’ll likely need to use the –BadItemLimit and –AcceptLargeDataLoss switches of the New-MoveRequest command to allow it to complete successfully. Be sure to weigh the risks of this action before you take it. Example of the New-MoveRequest command being used in a data loss scenario:

New-MoveRequest –Identity <mailboxname> -TargetDatabase <TargetDB>–BadItemLimit 200 –AcceptLargeDataLoss

In certain circumstances, even moving the mailbox may not be successful. I’ve encountered scenarios where the move request would fail because the mailbox kept being quarantined or some other unidentifiable error occured. While our options were limited in Exchange 2010, Exchange 2013/2016 gave us a very useful parameter in the New-MoveRequest command; the –ForceOffline parameter. Starting in Exchange 2010, mailbox moves were an online process where users could still access their mailbox while it was being moved. The –ForceOffline switch makes the move an offline process, similar to Exchange 2007 and older. I’ve found much success with using this command to overcome mailbox corruption issues during move requests. Using this switch will prevent a mailbox from becoming quarantined during the move, as well as keep the user from accessing an already corrupted mailbox. A common example of the command in action is as follows:

New-MoveRequest –Identity <mailboxname> -TargetDatabase <TargetDB>–BadItemLimit 200 –AcceptLargeDataLoss -ForceOffline

Now, should all of these recovery actions fail and the mailbox still will not move, the remaining options are to:

• Perform an Offline Defrag against the mailbox database holding the corrupted mailbox

• Export the contents of the corrupt mailbox to .PST using New-MailboxExportRequest

• Export the contents of the corrupt mailbox to .PST using an Outlook client

In each of the last two options, once the data is exported you should disable the mailbox, create a new mailbox, and import the data using either the New-MailboxImportRequest or an Outlook client.

Ease Your Transition to Office 365

While ENow is not a migration company, our software does enable IT Pros to assess their organization's readiness for Office 365 and accelerate their migration. In particular, ENow's solution helps organizations verify appropriate network connectivity, validate hybrid components (ADFS & AAD Connect) are setup correctly, and that mail routing is functioning.

For more information on how ENow can ease your transition to Office 365, please chat with the bot in the bottom right corner.