Fixing Search/Indexing issues within Exchange 2019 mailboxes.

In this blog, I will review and explain how to fix indexing issues within mailboxes hosted in Exchange 2019 and what are the main changes between indexing in Exchange 2013/2016 vs Exchange 2019.

Exchange 2013/2016 Search/Indexing

Exchange 2013 searching/indexing was based on Search Foundation, while Exchange 2016 was based on FAST search architecture, which was formerly used to index Sharepoint servers.

Both versions based on catalog indexing, which is among other components a folder located in the same folder where the EDB (database file) was.
The catalog folder size was up to 20% size of the actual database size.
The folder couldn’t be moved or located in a different drive and named by the GUID of the database that was indexed:

DB13 GUID

DB13 Catalog indexing folder

By the way, until Exchange 2016 CU3, in case of a failure with indexing at the passive copy, Exchange had to reseed the indexing from the active copy only.
This action took relatively a lot of networking bandwidth and it was especially critical between different geographical sites like production and DR sites.

Since Exchange 2016 CU3, the passive copy was able to rebuild its indexing from the local database itself, which saved a lot of bandwidth:

“High Availability Improvements

One of the challenging areas in some on-premises environment is the amount of data replicated with each database copy. In Exchange Server 2016 Cumulative Update 3, network bandwidth requirements between the active copy and passive HA copies are reduced. The Exchange Server Role Requirements Calculator has been updated to reflect these improvements. The local search instance reads data from a database copy on the local server, also known as “Read from Passive”. As a result of this change, passive HA copy search instances no longer need to coordinate with their active counterparts to perform index updates. Lagged database copies still coordinate with their active counterparts to perform index updates. This change also reduces database failover times when compared to Exchange Server 2013.”

Problem Statement

There are few cases where this “catalog folder” architecture might affect users experience and servers load:

• Rebuilding a failed search index might take a long time, in case users are working in Outlook Online mode or OWA, they will not be able to get any results in case of search queries.
• During the indexing rebuilding, eDiscovery and the Search-Mailbox activities also will not get any results, related to mailboxes located on the database where the catalog is getting fixed.
• Rebuilding the whole catalog indexing requires an intensive I/O and CPU resources from the server.
• Moving mailboxes between databases will cause reindexing the mailboxes again at the target databases and removing the moved mailbox indexing from the source database.
  This might be critical where mailboxes are having a very high number of items.
• Moving mailboxes will cause not only reindexing the target database copy but also all other copies of the same database.

Exchange 2019 Indexing

Exchange 2019 utilizes a new and different way of indexing.
It uses a new architecture based on the Big Funnel search engine, based on Bing technology which is already in use by Exchange Online as part of Office 365 in the cloud.
More information about Big Funnel can be found at the next link from Microsoft Ignite BRK3130:
https://www.youtube.com/watch?time_continue=871&v=VHrScskhCQk&feature=emb_logo

So, what is the big news, and how it solves the issues we reviewed earlier?

Well, the major change that arrived with Big Funnel, is that there no more one common catalog indexing per database which indexes the whole mailboxes within.
Now all indexing for each mailbox is located within the mailbox!

That means that

• Rebuilding a failed search index will be faster
• Moving mailboxes between databases will NOT cause reindexing the mailboxes again at the target and source databases since the moved mailbox already indexed!
• Moving mailboxes won’t cause reindexing all the copies since indexing is builtin inside the mailbox!

How to check indexing issues in Exchange 2019?

As I explained earlier, the indexing is builtin the mailbox.
To get more information about a specific mailbox indexing status, we can run the Get-MailboxStatistics cmd:

2 major attributes need to be examined to decide if there is an indexing issue with the mailbox:

BigfunnelNotIndexedSize
BigfunnelNotIndexedCount

In this case, we can see that 8 items are not being indexed.
To list all the mailboxes that are having items that are not getting indexed, we can run the next command.
Please notice that it can give us results only on mailboxes located on Exchange 2019!

Get-Mailbox -ResultSize Unlimited | Get-MailboxStatistics | ? {$_.BigfunnelNotIndexedCount -ge “1”} | ft DisplayName,BigfunnelNotIndexedCount

• Please note that Exchange 2019 CU5 fixed search issues with previous Exchange 2019 versions:
  “Cumulative Update 5 for Exchange Server 2019 is also required to fix a known issue with partial word searches when the client is using Outlook in online mode.”

How indexing is getting fixed in Exchange 2019?

To the current time May 2022, there are 3 main optional ways which indexing issues can be solved:

1. According to the video from Microsoft Ignite session BRK3130 you can use the Start-MailBoxAssistant command, which is available from Exchange 2019 CU11:

Start-MailboxAssistant -Identity <Mailbox id> -AssistantName BigfunnelRetryFeederTimeBasedAssistant

By the way, the Bigfunnel logs are located at the next path:

2. The second option is to use the next 2 commands from the Microsoft’s support web site Incomplete search results after installing an Exchange Server 2019 update (microsoft.com):

• New-SettingOverride -Name “Big Funnel Permanent Retry” -Component “BigFunnel”
  -Section “BigFunnelFailedItemsQuerySettings”
  -Parameters @(“ExcludePermanentFailures=false”)
  -Reason “Re-Index Messages” -MinVersion “15.2.922.7” -MaxVersion “15.2.986.22”

• Get-ExchangeDiagnosticInfo -Process Microsoft.Exchange.Directory.TopologyService -Component VariantConfiguration -Argument Refresh

3. The third way for fixing indexing issues within a mailbox is to move the mailbox to another database.
In this example, the mailbox which had 8 non-indexed items:

And it is located on DB03:

I then moved the mailbox to DB01:

After the mailbox was moved, it still has 2 unindexed items:

I guess that waiting some time would have solved the issue and reduce the unindexed items to 0, but since I wanted to decrease the time until there will be non-unindexed items, I have dismounted and mounted the database.

Please do NOT dismount a production database!
It will disconnect all the users from their mailboxes until the database will be mounted again!

Now there all the items in the mailbox are indexed!

Conclusions:

Exchange 2019 Big Funnel architecture and internal mailbox indexing, makes indexing issues much easier to fix and relatively very fast.