Following an upgrade to VMware NSX 4.1.x, index is out of sync

Products

• VMware NSX

Issue/Introduction

• Following an upgrade to VMware NSX 4.1.x, index is out of sync.
• On the NSX UI, some Inventory objects may appear to be missing e.g. Compute Manager, Groups etc.
• "start search resync all" has been executed on all the NSX Manager Nodes and the issue still persists.
• You see entries similar to the following in /var/log/search/search-inventory.log

2024-02-15T19:26:49.817Z ERROR pool-80-thread-1 UfoInitializer 85532 - [nsx@6876 comp="nsx-manager" errorCode="MP60525" level="ERROR" subcomp="cm-inventory"] [Search: Initialization] Exception while fetching metadata with id inventory org.opensearch.OpenSearchStatusException: OpenSearch exception [type=no_shard_available_action_exception, reason=No shard available for [get [inventory_metadata][_doc][inventory]: routing [null]]]

2024-02-15T19:56:33.970Z INFO pool-51-thread-1 UfoReIndexingManager 85532 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="cm-inventory"] [Indexing: Reindexing] Waiting for search framework initialization. Initialization status: FAIL

• 

• You may intermittently encounter the following error when accessing the Compute Managers section or performing related operations in the NSX Manager UI:

  “Compute Managers list failed: Index is currently out of sync, system is trying to recover.”

  This issue is not limited to upgrade scenarios—it can also occur during regular operation, without any recent changes to the environment

Environment

VMware NSX

Cause

The Search Framework Initialization fails in the inventory application, as the metadata cannot be read from Search after the upgrade due to the no_shard_available_action_exception.

Resolution

This issue is resolved in VMware NSX 4.2.0, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround:

Execute the below commands in root mode on all the NSX-T Manager nodes starting with the non-VIP leader nodes first.

1. service cm-inventory restart
2. service phonehome-coordinator restart
3. service idps-reporting-service restart
4. service proton restart

Alternatively, a rolling reboot of the NSX Manager VMs starting with the non-VIP leader nodes first can be performed.

After following the above work around steps, if the issue is not resolved in any one of the manager nodes and if the issue still persists with errors: "Index is currently out of sync, system is trying to recover. INDEXING_FAILURE (Error code: 60515)"

Then execute the following commands in the admin mode of that manager node to resolve the issue:

• start search resync manager
• start search resync all