Problem with Opensearch service showing unknown value in VMware Identity Manager (vIDM)

Products

• VMware Aria Suite

Issue/Introduction

Opensearch is showing unknown in the System Diagnostics Dashboard.

Opensearch - Health Unknown  
Opensearch - master node unknown  
Opensearch - indices count unknown  
Opensearch - docs count unknown  
Opensearch - unassigned shards unknown  
Opensearch - cluster nodes count unknown  
Opensearch - cluster nodes list unknown

The Opensearch process is trying to connect on port 9301 (SYN_SENT → node04:9301), which means it is expecting the peer to listen on 9301.

root@node02 [ ~ ]# sudo lsof -i :9301
COMMAND   PID       USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
java    30247 opensearch  172u  IPv4 #########      0t0  TCP master:52374->node04.example.com:9301 (SYN_SENT)
java    30247 opensearch  173u  IPv4 #########      0t0  TCP master:52342->node04.example.com:9301 (SYN_SENT)
java    30247 opensearch  174u  IPv4 #########      0t0  TCP master:52384->node04.example.com:9301 (SYN_SENT)
java    30247 opensearch  175u  IPv4 #########      0t0  TCP master:52386->node04.example.com:9301 (SYN_SENT)

We also see  RabbitMQ isn’t binding to 9300 on node04.

root@node04 [ ~ ]# sudo lsof -i :9300
COMMAND    PID       USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
epmd      2147       root    5u  IPv4     #####      0t0  TCP node04.example.com:epmd->node04.example.com:vrace (ESTABLISHED)
beam.smp  2616   rabbitmq   89u  IPv4     #####      0t0  TCP node04.example.com:vrace->node04.example.com:epmd (ESTABLISHED)
java     21278 opensearch  156u  IPv4 #########      0t0  TCP node04.example.com:43626->master:vrace (ESTABLISHED)
java     21278 opensearch  157u  IPv4 #########      0t0  TCP node04.example.com:43628->master:vrace (ESTABLISHED)
java     21278 opensearch  158u  IPv4 #########      0t0  TCP node04.example.com:43624->master:vrace (ESTABLISHED)

Environment

VMware Identity Manager 3.3.7

Cause

The OpenSearch cluster was not able to form, and the following error was observed on the problematic node:

• Command: curl http://localhost:9200/_cluster/state/nodes,master_node?pretty
• Seen in the results: {"error" : "MasterNotDiscoveredException[waited for [30s]]",  "status" : 503} 

Resolution

Perform the steps on all nodes.

1. Stop/Start the OpenSearch and the RabbitMQ in the sequence below: 

1. Stop the OpenSearch service: /etc/init.d/opensearch stop  
2. Stop the RabbitMQ service: systemctl stop rabbitmq-server.service 

3. Start the OpenSearch service: /etc/init.d/opensearch start

4. Start the RabbitMQ service: systemctl start rabbitmq-server.service 
      

2. Wait for some time and check if it joins the cluster, and check the cluster using:
http://localhost:9200/_cluster/state/nodes,master_node?pretty

 

Additional Information

Check if you are able to search for users or groups.
If not, please follow the KB: Unable to search certain users & groups in vIDM UI or assign users & groups to policies