NSX Manager UI inventory missing or not loading

Products

• VMware NSX

Issue/Introduction

• NSX UI does not load for System>Fabric>Hosts or Edges, T0 gateways, T1 gateways, or Home tab.
  
• Errors similar to below are displayed in NSX UI when trying to access the above locations

<DATE/TIME> : Error: Search index is out of sync for 'Host Transport Node'. Run the 'start search resync all' CLI command on the NSX appliance to resync. If the issue persists, contact VMware Support. (Error code: 60516):

Or

Error: Search index is out of sync. Run the 'start search resync all' CLI command on the NSX appliance to resync. If the issue persists, contact VMware Support. INDEXING_FAILURES_EXHAUSTED_RETRIES, params: [all] (Error code: 60516)

• Log lines similar to the below are encountered in /var/log/search/search-manager.log

ERROR UfoIndexer-BatchExecutor-search_policy-2 UfoGenericConverter #### - [nsx@#### comp="nsx-manager" errorCode="MP60511" level="ERROR" subcomp="manager"] [Indexing: DataProvider] Could not convert UFO object to Dto UfoObject{operationType=CREATE, descriptor=IndexingTypeDescriptor{tableName='HostTransportNode', streamTag=POLICY}, identifier=string_id: "/infra/sites/default/enforcement-points/default/host-transport-nodes/esxi##-#-########-####-####-####-############host-#######"

Environment

VMware NSX

Cause

If a communication error occurs between the NSX Manager and ESXi Transport Node during an uninstallation from NSX. This will keep the ESXi Transport Node in a marked_for_delete=true state in Corfu, which causes search indexing to return an error.

Resolution

This is a condition that may occur in a VMware NSX environment.

 

Workaround:

1. Force delete the ESXi Transport Node via API call, using the ESXi details found in the search-manager.log in the above Issue section:
   • Either using the path from the above error log: DELETE /policy/api/v1/infra/sites/default/enforcement-points/default/host-transport-nodes/esxi##-#-########-####-####-####-############host-#######?force=true
   • Or using the path actual ESXi Transport Node UUID: DELETE /api/v1/transport-nodes/########-####-####-####-############?force=true

If there are multiple hosts showing in the search-manager.log and causing the indexing to fail, repeat the above steps to delete them also.

If the workaround steps in this KB did not resolve the issue, please review the steps in this KB Installing NSX on Host fails with "already exists" error or upgrade fails with "Failed to get Host status"