Performing an elastic search on NSX UI may result in Index out of sync (Error code: 60516)
search cancel

Performing an elastic search on NSX UI may result in Index out of sync (Error code: 60516)

book

Article ID: 322457

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • If there was a NVDS to VDS migration recently and has been completed.
  • The following entries in the log /var/log/proton/nsxapi.log are found referencing the UUID of a transport node that does not exist:
2023-04-28T06:43:47.250Z WARN UfoIndexer-search_policy-0 TransportNodeQueryDao 5947 FABRIC [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] UUID 1d24d8ed-3c87-4c2e-849d-b0c33e336f3f could not be found
  • The following entries are found in log /var/log/search/search-manager.log
2025-01-30T22:15:18.482Z ERROR pool-1011-thread-2 UfoGenericConverter 7132 - [nsx@6876 comp="nsx-manager" errorCode="MP60511" level="ERROR" subcomp="manager"] [Indexing: DtoConversion] Could not convert UFO object to Dto by DTO converter UfoObject{operationType=CREATE, descriptor=IndexingTypeDescriptor{tableName='VdsTopology', streamTag=POLICY}, identifier=uuid {
  left: <left-key>
  right: <right-key>
}
2025-01-30T22:15:18.496Z  INFO pool-1011-thread-2 BulkContext 7132 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Adding IndexingTypeDescriptor{tableName='VdsTopology', streamTag=POLICY} to the indexing failure collection
  • When attempting to search in the NSX UI and the below alarm is observed in system alerts:
System is still in uncertain state please reload
  • When the search is executed, the below error is observed:
Error: Index out of sync, please resync via 'start search resync {0}'. (Error code: 60516)
  • There is no other functional impact.

Environment

VMware NSX-T Data Center 3.2.x

Cause

When doing an NVDS to VDS migration and a ESXi node was removed, this may result in a stale entry being present, which results in the Search index becoming out of sync.

Resolution

This is a known issue impacting VMware NSX.

Workaround:
If you were doing a NVDS to DVS migration and this migration is now complete, you can run the following API to clean up the stale entries.

POST https://<nsx-mgr>/api/v1/nvds-urt?action=cleanup 

Note: Please do not run the cleanup API if the NVDS to VDS migration is not complete.

 

If you believe you have encountered this issue and the noted workaround does not resolve it, please open a support case with Broadcom Support and refer to this KB article.

For more information, see Creating and managing Broadcom support cases.

Additional Information

If you are contacting Broadcom support about this issue, please provide the following:

  • NSX Manager support bundles
  • ESXi host support bundles for hosts that are failing to configure as transport nodes.
  • Text of any error messages seen in NSX GUI or command lines pertinent to the investigation

Handling Log Bundles for offline review with Broadcom support

Additionally, provide the output from the following commands run on the NSX Manager node as the root user:

curl -X GET http://localhost:9200/*_table_indexing_position/_search?q=hasExhaustedFailures:true&pretty -H 'Content-Type: application/json'
curl -X GET http://localhost:9200/*_table_indexing_position/_search?q=hasFailures:true&pretty -H 'Content-Type: application/json'

Powered by Wolken Software