Services on the vIDM cluster are down after restoring to snapshots and restarting the cluster nodes

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

In some cases, such as recovering from a failed patching attempt of vIDM 3.3.7 with CSP-97577, services in the cluster are down after restoring snapshots, and restarting the environment.

Symptoms:

• When running the command below from this KB, the output may show some of the nodes in the cluster have a status of down:
  • su root -c "echo -e 'password'|/opt/vmware/vpostgres/current/bin/psql -h localhost -p 9999 -U pgpool postgres -c \"show pool_nodes\""
• In the administrative console of the vIDM UI health page, services such as opensearch may display as Unknown or RED for some or all of the nodes.
• The /db/data directory on the secondary nodes are empty, or only contain a servicelog file.

Environment

Identity Manager 3.3.7

Resolution

1. Restart the services manually to put the cluster back in a healthy state, following the steps found in this KB.
2. For missing files in the /db/data directories of the secondary nodes, follow the steps found in this KB.
3. For opensearch health status showing as Unknown or RED in some or all of the nodes, follow the steps found in this KB.