"Index out of sync (Error code: 60516)" displayed when performing a search on NSX UI

Products

• VMware NSX

Issue/Introduction

• If there was a N-VDS to VDS migration recently and has been completed.
• The following entries in the log /var/log/proton/nsxapi.log are found referencing the UUID of a transport node that does not exist:

WARN UfoIndexer-search_policy-0 TransportNodeQueryDao 5947 FABRIC [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] UUID 1d24d8ed-####-####-####-b0c33e336f3f could not be found

• The following entries are found in log /var/log/search/search-manager.log

ERROR pool-1011-thread-2 UfoGenericConverter 7132 - [nsx@6876 comp="nsx-manager" errorCode="MP60511" level="ERROR" subcomp="manager"] [Indexing: DtoConversion] Could not convert UFO object to Dto by DTO converter UfoObject{operationType=CREATE, descriptor=IndexingTypeDescriptor{tableName='VdsTopology', streamTag=POLICY}, identifier=uuid {
  left: <left-key-uuid>
  right: <right-key-uuid>
}
INFO pool-1011-thread-2 BulkContext 7132 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Adding IndexingTypeDescriptor{tableName='VdsTopology', streamTag=POLICY} to the indexing failure collection

• When attempting to search in the NSX UI, the below alarm is observed in system alerts:

System is still in uncertain state please reload

• When the search is executed, the below error is observed:

Error: Index out of sync, please resync via 'start search resync {0}'. (Error code: 60516)

• There is no other functional impact.

Environment

VMware NSX-T Data Center 3.x

Cause

When doing an N-VDS to VDS migration and a ESXi node was removed, this may result in a stale entry being present, which results in the Search index becoming out of sync.

Resolution

This is a known issue impacting VMware NSX.

Workaround:
If you were doing a N-VDS to DVS migration and this migration is now complete, you can run the following API to clean up the stale entries.

POST https://<nsx-mgr>/api/v1/nvds-urt?action=cleanup 

Note: Please do not run the cleanup API if the N-VDS to VDS migration is not complete.

 

If you believe you have encountered this issue and the noted workaround does not resolve it, please open a support case with Broadcom Support and refer to this KB article.

For more information, see Creating and managing Broadcom support cases.

Additional Information

If you are contacting Broadcom support about this issue, please provide the following:

• NSX Manager support bundles
• ESXi host support bundles for hosts that are failing to configure as transport nodes.
• Text of any error messages seen in NSX GUI or command lines pertinent to the investigation

Handling Log Bundles for offline review with Broadcom support

• Collect Support Bundles for Troubleshooting VMware NSX
• Uploading files to cases on the Broadcom Support Portal
• Creating and managing Broadcom support cases

Additionally, provide the output from the following commands run on the NSX Manager node as the root user:

curl -X GET http://localhost:9200/*_table_indexing_position/_search?q=hasExhaustedFailures:true&pretty -H 'Content-Type: application/json'
curl -X GET http://localhost:9200/*_table_indexing_position/_search?q=hasFailures:true&pretty -H 'Content-Type: application/json'

If this KB did not help resolve your issue, you can review the following kb for further troubleshooting steps

Troubleshooting N-VDS to VDS migration