Unable to acknowledge/resolve/suppress NSX alarms "Failed to update the status for the selected alarm"

Products

• VMware NSX

Issue/Introduction

• NSX upgraded to 4.x.
• Unable to acknowledge/resolve/suppress NSX alarms: "Failed to update the status for the selected alarm"
• During the phonehome corordinator service restart, there was an issue with Search Framework Initialization.
• Search Framework failed to initialize because it can not read metadata from Opensearch because of the following exception in the var/log/phonehome-coordinator/phonehome-coordinator.log
  

  <timestamp> WARN pool-107-thread-1 UfoInitializer 75837 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="monitoring"] [Search: Initialization] Exception while metadata with id monitoring org.opensearch.OpenSearchStatusException: OpenSearch exception [type=no_shard_available_action_exception, reason=No shard available for [get [monitoring_metadata][monitoring]: routing [null]]]
          at org.opensearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:210) ~[opensearch-2.13.0.jar:2.13.0]
          at org.opensearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2235) ~[opensearch-rest-high-level-client-2.13.0.jar:2.13.0]

Environment

VMware NSX

Cause

The initialization failure of the Search Framework was traced back to faulty metadata indices originating from an older release. The issue is intermittent (dependent on system load during the read attempt). 

Resolution

Execute the following commands on all affected nodes.

1. SSH as the root user and restart the phonehome-coordinator service using the CLI:
   

   service phonehome-coordinator restart

2. Confirmed service initialization is completed from /var/log/phonehome-coordinator/phonehome-coordinator.log

   <time stamp>  INFO UfoReIndexingManager-Reindexing-0 UfoReIndexingManager 1236216 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="monitoring"] [Indexing: Reindexing] Search framework is initialized. Initialization status: DONE

3. Then SSH as the admin user and resync the search using CLI:
   

   start search resync telemetry