Timed out while syncing indexes. SYNC_TIMEOUT (Error code: 60514) encountered in NSX

Products

• VMware NSX

Issue/Introduction

• Error "Time out while syncing indexes. SYNC_TIMEOUT (Errorcode:60514)" is encountered in the appliance UI. 
  
• Notification "System is still in uncertain state, please reload" is displayed.
  
• Entries similar to the below will be encountered on an NSX local manager in /var/log/search/search-manager.log:

  ERROR pool-693-thread-1 SearchSyncManagerImpl 86855 - [nsx@6876 comp="nsx-manager" errorCode="MP60514" level="ERROR" reqId="########-####-####-####-########e2ea" subcomp="manager" username="system"] [QueryExecution: ConsistencyCheck] Timed out while waiting for search to sync up with corfu read position [SearchSyncState{stream='search_inventory', table='ComputeManagerStatusModelMsg', corfuPosition=3176383702}]

• Entries similar to the below will be encountered on an NSX local manager in /var/log/cm-inventory/cm-inventory.log:

  WARN http-nio-127.0.0.1-7443-exec-5 MpRestService 86657 SYSTEM [nsx@6876 comp="nsx-manager" level="WARNING" reqId="########-####-####-####-########e2ea" subcomp="cm-inventory" username="nsx-opsagent"] Failed to get extension certificate

• On an NSX Global Manager you might see entries similar to the below in /var/log/gmanager/gmanager.log:

  INFO http-nio-127.0.0.1-64440-exec-867 NsxBaseRestController 511737 SYSTEM [nsx@6876 audit="true" comp="global-manager" level="INFO" subcomp="global-manager"] UserName:'admin' ModuleName:'nsx-search' Operation:'POST@/api/v1/search/aggregate' Operation status: 'failure' Error: Timed out while syncing indexes.

• Performing "start search resync all" does not resolve the issue.
• You might see objects in the UI that have already been deleted.

Environment

VMware NSX

Cause

The system times out while syncing the indexes.

Resolution

This is a known issue impacting VMware NSX.

Workaround
It is possible to work around this issue by performing a reboot of the NSX managers one by one. Wait for the rebooted NSX Manager to be online before rebooting another.

1. Log in to the CLI of the NSX Manager.
2. Run the following command: 
   nsx-manager> reboot
Are you sure you want to reboot (yes/no): y