Error(Red X) on Integrated Components, Opensearch - master node shows "unknown"

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• VIDM Node (One of the Node in VIDM Cluster) has Error on "Integrated Components" 

     

• Fails to fetch Master Node: 

     

• In "analytics.log" error is seen as below: 

  com.vmware.idm.analytics.elasticsearch.JestHealthResultHandler - Failed to check health: {"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null} 
  com.vmware.idm.analytics.elasticsearch.ElasticSearchHttpStorageAdapter - Unable to check cluster health, retention policy skipped. java.lang.RuntimeException: java.util.concurrent.TimeoutException
      at com.vmware.vidm.common.async.CompletableFutures.block(CompletableFutures.java:91)

• Running the opensearch cluster healthcheck command, returns cluster_manager_not_discovered_exception

  curl -XGET http://localhost:9200/_cluster/health?pretty=true

  

Environment

VMware Identity Manager 3.3.7

Cause

The node may intermittently fail to identify or connect to the OpenSearch Manager Node. 

Resolution

Find the impacted node, can be seen in the System Diagnostic Dashboard as the problematic OpenSearch node or by reviewing logs:

more /opt/vmware/opensearch/logs/horizon.log

 

1. If only 1 node is impacted by the issue then a service restart should be sufficient:

   /etc/init.d/opensearch restart

2. If all nodes are impacted then review the OpenSearch configuration: 

   1. Review the configuration OpenSearch configuration yaml:

      cat /opt/vmware/opensearch/config/opensearch.yml

   2. Validate the parameter cluster.initial_master_nodes or cluster.initial_cluster_manager_nodes is configured to the same node across the vIDM cluster, which should be the first installed vIDM node.

   3. If parameter cluster.initial_master_nodes or cluster.initial_cluster_manager_nodes is different then stop opensearch on each node:

      /etc/init.d/opensearch stop

   4. Update the configuration yaml to have the same primary node across the cluster:

      vi /opt/vmware/opensearch/config/opensearch.yml

      Note: vi editor can be used with:
      • Key i or Insert key to change to editor mode
      • Escape key to leave editor mode
      • :w to save the change
      • :q the exit vi

   5. Start the service on the primary node first, then on each other node:

      /etc/init.d/opensearch start

   6. Validate the OpenSearch cluster health:

      curl -XGET http://localhost:9200/_cluster/health?pretty=true