Preforming an elastic search on NSX UI may result in Index out of sync (Error code: 60516)

Products

• VMware NSX

Issue/Introduction

Symptoms:

• You're running NSX 3.2.x.
• You did run an NVDS to VDS migration recently and this has completed.
  You observe the following entries in the log /var/log/proton/nsxapi.log referencing a UUID of a transport node that does not exist:

2023-04-28T06:43:47.250Z WARN UfoIndexer-search_policy-0 TransportNodeQueryDao 5947 FABRIC [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] UUID 1d24d8ed-3c87-4c2e-849d-b0c33e336f3f could not be found

• You're attempting to search on NSX UI and may observe the below alarm in system alerts when searching:

System is still in uncertain state please reload

• When the search completes, we may see the below error:

Error: Index out of sync, please resync via 'start search resync {0}'. (Error code: 60516)

• You observe no other functional impact.

 

Environment

VMware NSX-T Data Center

Cause

When doing an NVDS to VDS migration and a ESXi node was removed, this may result in a stale entry being present, which results in the Search index becoming out of sync.

Resolution

This is a known issue impacting VMware NSX.

Workaround:
If you were doing a NVDS to DVS migration and this migration is now complete, you can run the following API to clean up the stale entries.

POST https://<nsx-mgr>/api/v1/nvds-urt?action=cleanup
 

Note: Please do not run the cleanup API if the NVDS to VDS migration is not complete.

If the above does not resolve the issue, please run the below commands on the NSX manager CLI as root user:

curl -X GET http://localhost:9200/*_table_indexing_position/_search?q=hasExhaustedFailures:true&pretty -H 'Content-Type: application/json'

curl -X GET http://localhost:9200/*_table_indexing_position/_search?q=hasFailures:true&pretty -H 'Content-Type: application/json'
 

Record the output of the above commands.
Collect the support bundle for NSX managers taking note of the time stamps when the above API calls where run and open a support request with Broadcom Support and refer to this KB article.