"Index out of sync (Error code: 60516)" displayed when performing a search on NSX UI

Products

VMware NSX

Issue/Introduction

An N-VDS to VDS migration was recently completed.

The following entries in the log /var/log/proton/nsxapi.log are found referencing the UUID of a transport node that does not exist:

WARN UfoIndexer-search_policy-0 TransportNodeQueryDao 5947 FABRIC [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] UUID 1d24d8ed-####-####-####-b0c33e336f3f could not be found

The following entries are found in the log /var/log/search/search-manager.log:

ERROR pool-1011-thread-2 UfoGenericConverter 7132 - [nsx@6876 comp="nsx-manager" errorCode="MP60511" level="ERROR" subcomp="manager"] [Indexing: DtoConversion] Could not convert UFO object to Dto by DTO converter UfoObject{operationType=CREATE, descriptor=IndexingTypeDescriptor{tableName='VdsTopology', streamTag=POLICY}, identifier=uuid {
  left: <left-key-uuid>
  right: <right-key-uuid>
}
INFO pool-1011-thread-2 BulkContext 7132 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Adding IndexingTypeDescriptor{tableName='VdsTopology', streamTag=POLICY} to the indexing failure collection

When you search in the NSX UI, the following alarm is observed in system alerts:
```
System is still in uncertain state please reload
```

When the search is executed, the following error is observed:

Error: Index out of sync, please resync via 'start search resync {0}'. (Error code: 60516)

There is no other functional impact.

Environment

VMware NSX

Cause

If an ESXi node is removed during an N-VDS to VDS migration, a stale entry might remain and cause the search index to become out of sync.

Resolution

This is a known issue impacting VMware NSX.

Warning: Confirm NSX Manager backups are complete. Run this process during a scheduled maintenance window.

Workaround:

If you completed an N-VDS to DVS migration, run the following API only once to clean up the stale entries:

Run the API call directed to the VIP or one individual NSX Manager:
```
POST https://<nsx-mgr>/api/v1/nvds-urt?action=cleanup
```

Alternatively, run the API via curl on only one NSX Manager node CLI as root:

curl -l -k -u 'admin' -H 'Content-Type:application/json' -X POST https://localhost/api/v1/nvds-urt?action=cleanup

Note: Do not run the cleanup API if the N-VDS to VDS migration is not complete.

If this workaround does not resolve the issue, open a support case with Broadcom Support.

For more information, see Creating and managing Broadcom support cases.

Additional Information

If you contact Broadcom Support about this issue, provide the following:

NSX Manager support bundles.
ESXi host support bundles for hosts that are failing to configure as transport nodes.
Text of any relevant error messages seen in the NSX UI or command line.

Handling log bundles for offline review with Broadcom Support:

Additionally, provide the output from the following commands run on the NSX Manager node as the root user:

curl -X GET http://localhost:9200/*_table_indexing_position/_search?q=hasExhaustedFailures:true&pretty -H 'Content-Type: application/json'
curl -X GET http://localhost:9200/*_table_indexing_position/_search?q=hasFailures:true&pretty -H 'Content-Type: application/json'

If this article did not help resolve your issue, review the following article for further troubleshooting steps:

Troubleshooting N-VDS to VDS migration