Following an upgrade to VMware NSX 4.1.x, index is out of sync

Products

• VMware NSX

Issue/Introduction

• Following an upgrade to VMware NSX 4.1.x, index is out of sync.
• On the NSX UI, some Inventory objects may appear to be missing e.g. Compute Manager, Groups etc.
• "start search resync all" has been executed on all the NSX Manager Nodes and the issue still persists.
• You see entries similar to the following in /var/log/search/search-inventory.log

2024-02-15T19:26:49.817Z ERROR pool-80-thread-1 UfoInitializer 85532 - [nsx@6876 comp="nsx-manager" errorCode="MP60525" level="ERROR" subcomp="cm-inventory"] [Search: Initialization] Exception while fetching metadata with id inventory org.opensearch.OpenSearchStatusException: OpenSearch exception [type=no_shard_available_action_exception, reason=No shard available for [get [inventory_metadata][_doc][inventory]: routing [null]]]

2024-02-15T19:56:33.970Z INFO pool-51-thread-1 UfoReIndexingManager 85532 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="cm-inventory"] [Indexing: Reindexing] Waiting for search framework initialization. Initialization status: FAIL

Environment

VMware NSX 4.x

Cause

The Search Framework Initialization fails in the inventory application, as the metadata cannot be read from Search after the upgrade due to the no_shard_available_action_exception.

Resolution

This issue is resolved in VMware NSX 4.2.0

Workaround:

Execute the below commands in root mode on all the NSX-T Manager nodes starting with the non-VIP leader nodes first.

1. service cm-inventory restart
2. service phonehome-coordinator restart
3. service idps-reporting-service restart
4. service proton restart

Alternatively, a rolling reboot of the NSX Manager VMs starting with the non-VIP leader nodes first can be performed.

After following the above work around steps, if the issue is not resolved in any one of the manager nodes and if the issue still persists with errors: "Index is currently out of sync, system is trying to recover. INDEXING_FAILURE (Error code: 60515)"

Then execute the following commands in the admin mode of that manager node to resolve the issue:

• start search resync manager
• start search resync all