Exchange Monitoring: DAG Best Practices

On-premises Exchange servers are still a thing, and with future versions of Exchange coming on-premises we can assume they still will be for some time to come and on-premises Exchange monitoring is recommended. If your organization still runs on-premises Exchange servers, then Datacenter Activation Coordination (DAC) is a feature you need to understand.

According to Microsoft, DAC mode is "used to control the database mount on startup behavior of a DAG. This control is designed to prevent split brain from occurring at the database level during a datacenter switchback. Split brain, also known as split brain syndrome, is a condition that results in a database being mounted as an active copy on two members of the same DAG that are unable to communicate with one another. Split brain is prevented using DAC mode, because DAC mode requires DAG members to obtain permission to mount databases before they can be mounted."

In this blog post, I will explain what DAC mode is, how it works, and how it should be used for on-premises Exchange deployments.

When to turn on DAC mode

DAC is designed to avoid split-brain, or a situation where two or more different copies of a mailbox database are running independently. Without DAC mode turned on, it is possible for a datacenter outage to result in a situation where different copies of your mailbox databases are running, which in turn means you can have different copies of your mailbox databases with different information in each.

This split-brain syndrome is more likely to occur when your datacenters suffer a network outage that prevents DAG members from communicating with each other. If you have a DAG setup with copies of any single database in more than one datacenter, you should turn on DAC mode to prevent a split-brain situation.

If you have a DAG with two servers and a file share witness in your main datacenter, and two servers in a secondary datacenter, then DAC mode is for you.

How DAC mode works

Turing on DAC mode enables a protocol called Datacenter Activation Coordination Protocol (DACP). When DAC mode is activated, databases will not automatically mount (even if they have quorum) unless two specific conditions are met.

When DAC mode is turned on (DAC mode must be manually activated, it is never on by default) the Active Manager (which is responsible for starting databases) holds a DACP bit in memory. At startup, this DACP bit is set to 0 and will not allow any databases on that server to start unless the DACP bit is set to 1. The Active Manager will only switch its DACP bit to 1 under one of two circumstances.

• If the Active Manager can communicate with all servers that are members of the DAG
• If the Active Manager can communicate with another server with its DACP bit set to 1

While that seems simple, it can be a confusing to wrap your head around. Let’s look at another way to explain DAC with the human DAG demonstration.

The Human DAG

Back when the Master program was a thing, one of the program instructors had a live demonstration he would use to demonstrate how DAC mode works.

The “Human DAG” demonstration became somewhat of a fixture in the Exchange Master program, and I am pretty sure all of us who took that course did it. This little exercise was a great way to make it very clear how DAC mode works.

Available on my wesbite, I have a recording of the Human DAG presentation you can watch. Please excuse how haggard we all look in the video. It was a rough month of 12-16 hours a day in the classroom.

Datacenter Switchovers with DAC

Another awesome benefit of having DAC mode turned on is that it makes datacenter switchovers much easier.

When you have DAC mode turned on, you can use the following PowerShell cmdlets for a datacenter switchover.

• Stop-DatabaseAvailabilityGroup
  
  
• Restore-DatabaseAvailabilityGroup
  
  
• Start-DatabaseAvailabilityGroup

The process for safely doing a datacenter switchover without DAC mode is considerably more complicated. I’m not going to take the space to detail the whole process here, but you can find the process detailed in "Datacenter Switchovers" documentation from Microsoft.

Turning on DAC mode

Turning on DAC mode is simple. All you need is a single PowerShell command.

            Set-DatabaseAvailabilityGroup -Identity DAG1 -DatacenterActivationMode DagOnly

The options for the -DatacenterActivationMode switch are either “DagOnly” or “Off”. Personally, I would have made that switch be “On” or “Off” but we work with the PowerShell we have.

The downside to DAC mode

The reason that DAC mode is not on by default when you create a multi-site DAG is it can prevent your Exchange databases from coming online. Proper Exchange monitoring can help detect DAG issues. That is the point of this feature, but I have seen customers who intentionally turned-on DAC mode and then were frustrated that it kept their databases off-line.

If you turn on DAC mode, you should understand that the point of this configuration is to keep your databases off-line when all servers in the DAG cannot communicate with each other.

Wrap up

If you have an on-premises Exchange deployment with a DAG setup in multiple datacenters, then you should strongly consider turning on DAC mode. If you do, please be aware that DAC mode can and will keep databases from mounting, even when you think those databases should be coming on-line.

Exchange Hybrid and Office 365 Monitoring and Reporting

On-premises components, such as AD FS, PTA, and Exchange Hybrid are critical for Office 365 end user experience. In addition, something as trivial as expiring Exchange or AD FS certificates can certainly lead to unexpected outages. By proactively monitoring hybrid components, ENow gives you early warnings where hybrid components are reaching a critical state, or even for an upcoming expiring certificate. Knowing immediately when a problem happens, where the fault lies, and why the issue has occurred, ensures that any outages are detected and solved as quickly as possible.

Watch all aspects of your Exchange environment from a single pane of glass: client access, mailbox, and Edge servers; DAGs and databases; network, DNS, and Active Directory connectivity; Outlook, ActiveSync, and EWS client access.

Access your free 14-day trial of ENow’s Exchange Hybrid and Office 365 Monitoring and Reporting today!