VMware Identity Manager postgres cluster outage due to loss of delegate IP

Products

• VMware Aria Suite

Issue/Introduction

The purpose of this article is to Remediate VMware Identity Manager Cluster health.

Symptoms:
VMware Identity Manager cluster health status displays as CRITICAL in vRealize Suite Lifecycle Manager Health Notification due to network loss in the VMware Identity Manager appliance.

Note: This article is applicable for VMware Identity Manager cluster deployed through vRSLCM.

Command : /usr/local/bin/pcp_watchdog_info -p 9898 -h localhost -U pgpool

Above command would prompt for a password. If /usr/local/etc/pgpool.pwd file is present on the VMware Identity Manager node, that would contain the password. Else, use the default password - 'password'.

Command parameters help :

-h : The host against which the command would be run, here its 'localhost'.
-p : Port on which pgpool accepts connections, which is 9898
-U : The Pgpool health check and replication delay check user, which is pgpool

Expected Response :

3 YES <Host1>:9999 Linux <Host1> <Host1>

<Host1>:9999 Linux <Host1> <Host1> 9999 9000 4 MASTER
<Host2>:9999 Linux <Host2> <Host2> 9999 9000 7 STANDBY
<Host3>:9999 Linux <Host3> <Host3> 9999 9000 7 STANDBY

In the response, there needs to be a MASTER node and 2 STANDBY nodes present. If any of the node's status is SHUTDOWN or DEAD or the command execution is struck, follow the below steps to resolve the issue.

Environment

VMware Identity Manager 3.3.x

Cause

Network loss on the postgres cluster primary node.

Resolution

Currently, there is no resolution.

Workaround:
To workaround this issue, follow the below steps :

1. Gracefully bring down the services on VMware Identity Manager nodes following the steps in Graceful Shutdown and Power On of a VMware Identity Manager PostgreSQL cluster
2. Power OFF the VMware Identity Manager appliances in vCenter.
3. Power ON VMware Identity Manager nodes through vRealize Suite Lifecycle Manager.