A career-defining moment for many IT professionals is the day that their customer requests some data to be restored from backup, and the backup is found to be not working. Or even worse, the backup doesn’t even exist.
It’s not a situation you ever want to find yourself in, and to help you to avoid it I’m going to tell you about the common (but easily remedied) mistakes that can lead to such an unfortunate position.
Or, as I like to call them, the 7 Deadly Sins of Exchange Server Backups.
Microsoft Exchange Server uses a transactional database system that is almost continuously in a state of change with a memory buffer, log files, and the database file itself all working together in unison to maintain the integrity and consistency of the mailbox data.
What this means for Exchange Server backups is that you can’t simply take a backup of the Exchange data by treating it as a bunch of files on a disk. Even though that may yield you a set of files on backup media, recovery from those files (without the risk of significant data loss) is impossible.
Microsoft only supports application-aware, VSS-based backups that properly handle the Exchange database engine and take a consistent backup of the database and transaction log files. If you aren’t sure, go and check that your chosen backup product is supported for Exchange Server backups.
I frequently encounter customers who were not aware that they were not taking application-aware backups of their Exchange servers, but did notice that the database transaction logs on the server would accumulate and consume all of the available disk space on their log volume. A proper backup of Exchange will truncate those log files, which solves the issue of disk consumption.
Unfortunately, when the customers search online for a solution to their disk space problem on the log volume they often find advice to enable circular logging for the database. Circular logging automatically removes transaction logs from the server as changes are committed to the database, so it avoids the disk space problem. But circular logging can mask the fact that the Exchange databases are not being backed up, and has a serious impact on the recoverability of the database as well.
As a general rule circular logging does not need to be used when the Exchange databases are being backed up correctly. The exception to this is when the Exchange database are replicated in a database availability group (DAG) and you are using Native Data Protection instead of traditional backups.
Backup software (and backup administrators) can sometimes lie. I have seen many occasions when a backup report says that Exchange was successfully backed up when it actually has not been backed up at all.
Aside from malfunctioning backup software (or admins) it’s easy in larger IT environments for someone to create a new database on an Exchange server and neglect to add it to the backup jobs.
You can independently check whether your Exchange databases are backing up by running the Get-MailboxDatabase cmdlet.
Get-MailboxDatabase –Status | Select Name,Last*Backup,*InProgress
Even better, you can automate this check and get a daily report in your inbox by using my Get-DailyBackupAlerts.ps1 script.
Working backups are only half of the story. When it comes time to restore some data you don’t want to be left scratching your head trying to sort through vendor documentation to work out how to actually perform the restore in your environment.
For your Exchange Server backups there are multiple recovery scenarios that you should have clearly documented and tested:
The first two scenarios not only test your recovery procedures, but also the integrity of your backup sets. If you’ve never restored a mailbox or database from backup, how do you even know that the data sitting on disks or tapes is any good?
The last two scenarios are high impact and are not the sort of thing you can easily perform in a live, production environment. However, you can still document and test the procedure in a separate test lab environment so that you’re ready when disaster strikes.
Many organizations virtualize their Exchange servers, and use backup products that integrate with their virtualization platform to back up the virtual machines themselves.
It’s important to make sure that your VM backup product is taking application-aware backups of the Exchange server (see Sin #1 above), but another equally important point is that restoring VMs from snapshots is not supported by Microsoft. Rolling back a VM using a snapshot can have disastrous impacts on the mailbox databases, as well as other components of Exchange such as the transport database.
If you do need to recover data from a VM snapshot, use the recovery process documented by the backup product vendor. Typically, this involves using a specialized tool that they provide to extract the desired mailbox data from a backup set, and then ingesting it into the live database.
There are basically three groups of data that you need to protect for your Exchange server:
Your Exchange database backups cover the first group, but the other two are a little more complicated. Most of the Exchange server configuration is stored in Active Directory, so it will be protected by your AD backups and will be re-applied to Exchange if you ever need to do a recovery install.
However, some of the Exchange configuration relies on operating system components such as IIS and the certificate store. These are not stored in Active Directory, nor are they included in the Exchange database backups.
In an effort to reduce backup sizes some customers do not back up the Exchange server’s operating system volume and system state. In these cases, it’s important to make sure you’ve documented, automated, or backed up any additional operating system components that Exchange relies on, so that they can be reapplied quickly and easily in a recovery scenario.
I once worked with a customer that had an unfortunate datacenter outage caused by water leakage from the air-conditioning units. The water caused an electrical fault, which resulted in some smoke, that then triggered the fire suppression systems. This caused serious damage to the server hardware, and also to the storage system that was used for their disk-based backups. It also destroyed some backup tapes that had been stored on a shelf in the datacenter.
Fortunately for this customer they also had replicas of those tapes held by an off-site storage service, because the customer subscribed to the 3-2-1 Rule:
These days, off-site storage can be achieved without physically transporting backup tapes, thanks to the evolution of cloud-based storage. With sufficient network bandwidth and a backup or storage system that supports it, your backup sets can be continuously replicated to cloud storage to protect them from local disasters such as the one my customer experienced.
As you can see there are many ways that Exchange Server backups can go wrong. Hopefully you have read through the list and checked your own backups against it. If you aren’t committing any of those 7 deadly sins, then you’re doing better than what is sadly a large number of other customers out there. But if you have spotted a few weaknesses in your Exchange Server backups, it’s time to sort them out before you have your own career-defining moment.