Clearing AutoComplete and Other Recipient Caches Redux

This information can be stored in multiple locations, depending on the client used. Before Outlook 2010, this information was stored in local NK2 files. For Outlook 2010 and up, this information is stored in a hidden item in your mailbox. This item, the AutoComplete Stream, will be cached locally by Outlook for performance reasons. Outlook on the web (OWA) on the other hand, utilizes its own caching mechanism, storing this recipient information in a different part of your mailbox. In addition, other recipient information not directly stemming from addressing is stored as well, such as the Suggested Contacts folder.

The issue arises when mailboxes are moved to other organizations such as during a merger or acquisition, or other form of cross-Exchange organization or tenant mailbox move. When using an Exchange-related recipient, Outlook will store the underlying address that is used by Exchange – the infamous X500 legacyExchangeDN, which looks like /o=Organisation/ou=Administrative Group/cn=Recipients/cn=Username – together with the recipient. When picking an address from the list, you are effectively picking the associated x500 address.

Now, because AutoComplete and other recipient caches are static, this x500 address may become invalid at some point. That is, unless it has been applied to the migrated mailbox and thus can be resolved in the new environment. For example, in Exchange hybrid deployments, Active Directory Connect (DirSync) will take care of properly propagating this attribute, stamping it as x500 address on the proxyAddresses attribute.

However, not all migrations or M&A scenarios allow for this information to retained, it might get overlooked, or perhaps organizations wish to start with a clean slate. In that case, zapping this cached information on the back-end might be an option, as opposed to instructing end users to remove entries from the picker. Here is where the Clear-AutoComplete script comes in, which might support those scenarios. Depending on the scenario, this script can zap the desired information, clearing cached information with potentially outdated information. Downside is that end users will have to repopulate their recipient pick lists, which is perhaps a small price when compared to the potential non-deliveries and support calls that might be the result of not clearing this information.

Before we talk about using the script, first a bit on the various caching mechanisms . . . .

AutoComplete Stream

The autocomplete stream contains information from previously used recipients from Outlook (for Desktop). The list contains address information such as SMTP or LegacyDN addresses and is limited to 1,000 most-recently used entries. The information is stored in the so-called folder-associated information (FAI) table of your Inbox, which is a hidden folder. FAI entries can be inspected using utilities like MFCMAPI or EWS Editor, like in the screenshot below, showing the AutoComplete Stream using MFCMAPI.

As indicated, the autocomplete stream is stored an item of message class IPM.Configuration.Autocomplete. The entries themselves are stored in the property PR_ROAMING_BINARYSTREAM (pidTagRoamingBinary, tag 0x7C09000A) which, as the name implies, is in binary format. However, you can inspect the information in a more comprehensible format using MFCMAPI. In MFCMAPI, open up the PR_ROAMING_BINARYSTREAM property of the IPM.Configuration.Autocomplete item using the Smart View structure definition Nickname Cache.

Outlook will only retrieve the autocomplete stream when addressing its functionality, such as entering a recipient in the To section when composing an e-mail. Any updates to the autocomplete stream will not immediately be written back to the FAI item, but will be saved when Outlook is closed. Because of this, Outlook must be closed when making changes to the autocomplete stream in the mailbox, or Outlook may overwrite it with its own cached copy.

Note that Outlook caches autocomplete streams locally in %userprofile% under \AppData\Local\Microsoft\Outlook\RoamCache. The files are named Stream_Autocomplete*.dat, with only one .dat file per configured account. These files are used when running Outlook in Cached Mode and serve as a storage location for updating the FAI item.

End users can reset the nickname cache from Outlook via File > Options > Mail > Send Messages using the button Empty Auto-Complete List. Alternatively, the autocomplete list may be reset using a switch when starting Outlook, e.g. outlook.exe /CleanAutoCompleteCache.

Outlook on the web AutoComplete

Outlook on the web (OWA) caches recently used recipient information. Unfortunately, it does not share this information with Outlook. OWA stores its autocomplete stream information in the FAI table of the root folder of your mailbox, in an IPM.Configuration.OWA.AutocompleteCache message class item. In the image below, an example is shown from the associated contents of the root container of a mailbox, using MFCMAPI.

The autocomplete stream is stored in the item of message class IPM.Configuration.OWA.AutocompleteCache, but when accessing item through Exchange Web Services, you need to drop the IPM.Configuration prefix. The entries themselves are stored in the property PR_ROAMING_XMLSTREAM (pidTagRoamingXmlStream, tag 0x7c080102) which, as the name implies, is stored in XML format (in the screenshot the stream does not contain any entries).

Users using the Light version of Outlook WebApp can also reset OWA's recipient cache via Options > Messaging > Clear Most Recent Recipients List.

Suggested Contacts

The Suggested Contacts folder is created by Outlook. It is a folder to which contact objects will be added for each recipient used to send messages to, but that are not present in your Contacts folder. Like ordinary address books, Suggested Contacts is used as an additional address book for looking up recipients.

Note that unlike autocomplete, Suggested Contacts entries cannot be searched when typing the first few characters of a recipient. Also, users cannot manually add entries to the Suggested Contacts.

Recipient Cache

The Recipient Cache folder got introduced with Exchange 2013. It is a non-visible, well-known folder that sits below the Contacts folder, as shown in the MFCMAPI screenshot below. The specific purpose of the Recipient Cache folder is not documented. However, it does contain cached recipient information.

Clean-AutoComplete Script

The Clean-AutoComplete script is straightforward. It uses Exchange Web Services, and requires the EWS Managed API (how to). The mailbox can be hosted in Exchange on-premises or Exchange Online. The Clear Autocomplete script itself can be downloaded from GitHub. It can use Basic Authentication or Modern Authentication (OAuth2, requiring the NuGet package Microsoft.Identity.Client). For the latter, you need to register an application in Azure AD, configure it with sufficient permissions to access the mailboxes, and provide admin consent.

Usage of Clean-AutoComplete.ps1 is as follows:

Clear-AutoComplete.ps1 [-Identity] String[] [-Server <String>] [-Impersonation] [-Credentials<PSCredential>] [[-CertificateThumbprint <String>] [-CertificateFile <String> [-CertificatePassword ] <SecureString> [-Secret <SecureString>]] -TenantId <String> -ClientId <String>] [-Type <Array>] [-Pattern <String[]>] [-WhatIf] [-Confirm]

Where:

• Identity is the identity or e-mail address of one or more mailboxes.
• Server is the name of the Exchange Web Services endpoint, e.g. outlook.office365.com. When omitted, the script will try using AutoDiscover to discover the endpoint.
• Switch Impersonation specifies if impersonation will be used for mailbox access. This is required for Modern Authentication.
• Type specifies what cached recipient information to clear. Options are Outlook, OWA, SuggestedContacts, RecipientCache or All for all of these. Default is Outlook,OWA.
• Pattern can be used to specify a pattern of addresses to remove, and only works for the non-binary OWA autocomplete, SuggestedContacts or RecipientCache. Usage is as follows: -Pattern '*@contoso.com' or '/O=Contoso/*'.
• TrustAll tells the script to accept any certificate, self-signed included.
• TenantId specifies the ID of the Tenant when using a mailbox hosted in Exchange Online.
• ClientId to specify the Application ID of the registered application in Azure Active Directory.
• Credentials to specify the Basic Authentication credentials for on-premises usage or against Exchange Online when OAuth is not an option.
• CertificateThumbprint is the thumbprint of the certificate to use for modern authentication. The certificate with the public key needs to stored with the registered application for authentication. The certificate with the private key should be present in the local certificate store.
• CertificateFile and CertificatePassword to specify the file of the certificate to use. The file shoud contain the private key, and the password to unlock the file can be specified using CertificatePassword. Note that passed CertificatePassword needs to be a Secure String.
• Secret can be used to specify the secret to authenticate using the registered application, by providing a Secure String.

Note that you can only specify one authentication method. So, when specifying Credentials, you cannot specify one of the modern authentication parameters such as CertificateFile. Also, when specifying CertificateThumbprint, you cannot – and need not - specify Secret nor Credentials.

For example, assume you have created a self-signed certificate to use with the script. You want to use OAuth2, and have registered an application for it in Azure AD and configured sufficient permissions such as Exchange.ManageAsApp. To remove the fabrikom.com entries from the OWA cache, you can run the script as follows:

.\Clean-AutoComplete.ps1 -Identity id@consoto.com -Server outlook.office365.com -TenantId <TenantID> -ClientId <AppId> - Secret (ConvertToSecureString -String 'MySecret' -Force -AsPlainText) -Type OWA -Pattern '*@contoso.com' -Verbose

We added the Verbose switch to display additional information during processing. Also, you can suppress the confirmation prompt using the common -Confirm:$False.

Planning a Migration to Office 365?

Accelerate Your Transition and Reduce Risk with ENow’s Mailscape!

Monitoring every key aspect at every single turn may seem impossible, which is why it is always safer, smarter, and less complicated if you find the right systems management tool. But why pay for an expensive solution that only assists you during your migration and then becomes obsolete? Why not purchase an affordable, intuitive software that can help during each phase of your migration and beyond?  Enow's Mailscape is an affordable, intuitive and award-winning software solution that can help during each phase of your migration and, more importantly, help you beyond your migration.

Planning Phase

• Discover and clean up: Utilize Mailscape’s built in 250 reports to identify inactive mailboxes, distribution lists, mobile devices, client versions and public folders.
• Ensure smooth transition for Senior Management: Easily identify delegate permissions to ensure executives are not negatively impacted.
• Solidify your design: Obtain all the statistics you need to plan out and create your new storage design, including existing database size, current mailbox distribution, white space statistics and storage trending.

Deployment Phase

• Validate and test your new Exchange servers.
• Automatically test DAG configuration and failover, internal and external mail flow, CAS Connections, and hybrid connectivity.

Migration Phase

• Migration monitoring: Moving large amounts of data can cause disk outage, as large amounts of log files are created during the migration process. Mailscape helps monitor both the source and target servers to ensure all key systems are operational and do not fail.
• Disk space monitoring: ensures that log files do not fill up either source or target servers.
• Resource monitoring: assists in keeping an eye on CPU and memory usage.
• Mail flow monitoring: alerts if mail stops flowing on the target system.
• Backup monitoring: notifies you if any of your databases have not been backed up.

Access your free 14-day trial of ENow’s Office 365 Monitoring and Reporting solution today!