NSX Manager alarm "Search index is out of sync for 'Alarm'" (Error code: 60516)

Products

• VMware NSX

Issue/Introduction

In VMware NSX, a manager node may report an alarm regarding the search index synchronization. You will see the following error message in the NSX Manager UI or logs: Error: Search index is out of sync for 'Alarm'. Run the 'start search resync all' CLI command on the NSX appliance to resync. (Error code: 60516). In some instances, running the recommended CLI command does not resolve the alarm.

Environment

 

• VMware NSX-T Data Center

• VMware NSX 4.x

 

Cause

This issue occurs when the internal search index used by the NSX Management Plane becomes out of sync with the actual data in the database. If the start search resync all command fails, it often indicates the search service requires a full restart.

Resolution

If the start search resync all command does not clear the alarm, you must reboot the affected NSX Manager(s):

1. Perform a graceful reboot of one affected NSX Manager appliance.
2. After the reboot, log back into the NSX Manager UI.
3. Verify that all NSX services on the rebooted Manager have fully initialized and that the cluster status is healthy and stable.
4. Verify whether the alarm has cleared. Note that it may take several minutes for services to fully stabilize.
5. If additional NSX Managers require a reboot, wait for the cluster to stabilize before rebooting the next Manager. Reboot Managers one at a time (rolling reboot) to maintain cluster stability.
6. Repeat the process for any remaining affected NSX Managers.

Additional Information

Subscribe to this knowledge article to get updates on this issue.