Error: "Pgpool cluster is unhealthy" During vIDM 3.3.7 Patching (KB 412021 Part 1, Step 7)

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• When attempting to run the following command, as described in Step 7 of Part 1 in KB 412021: 

  ./CSP-102092-patch-automation.sh -f CSP-102092-Appliance-3.3.7.zip -r

• You encounter the following error:

  [ERROR] Pgpool cluster is unhealthy. Unable to proceed with the patching.

  

• Checking Postgres status shows that one or more of the VIDM nodes are down.  You can use the following KB as a guide on how to check
  Postgres status on all VIDM nodes Troubleshooting VMware Identity Manager postgres cluster deployed through Aria Suite Lifecycle (vRSLCM) 

  node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay | last_status_change
  ---------+---------------+------+--------+-----------+---------+------------+-------------------+-------------------+---------------------
   0 | Host1 | 5432 | up | 0.333333 | primary | 0 | false | 0 | 2019-10-14 06:05:42
   1 | Host2 | 5432 | down | 0.333333 | standby | 0 | false | 0 | 2019-10-14 06:05:42
   2 | Host3 | 5432 | down | 0.333333 | standby | 0 | true | 0 | 2019-10-14 06:05:42
  (3 rows)

• VMware Aria Suite Lifecycle was patched before VIDM was patched and day 2 operations were applied. 

 

Environment

• VMware Identity Manager 3.3.7

Cause

• Before proceeding with the patching process, the automation script performs a health check on the Pgpool cluster. The Pgpool cluster is a critical component of the VMware Identity Manager (vIDM) PostgreSQL database in a clustered deployment. If the cluster is not healthy, the patching process will be halted, and this error will be displayed.

Resolution

• Typically, this error indicates that the VMware Identity Manager cluster health status is reported as CRITICAL in your Aria Suite Lifecycle Health Notification.

• To resolve this issue, follow the steps outlined in Troubleshooting VMware Identity Manager postgres cluster deployed through Aria Suite Lifecycle (vRSLCM) to restore the cluster's health. 

• Once the cluster health is restored, reattempt the patching process.

 

Note: The [ERROR] Pgpool cluster is unhealthy. Unable to proceed with the patching. can also trigger if the Cluster Auto-Recovery is enabled for vIDM/globalenvironment in Aria Suite Lifecycle > Lifecycle Operations > Environment. Disable Cluster Auto-Recovery is it is enabled from Aria Suite Lifecycle > Lifecycle Operations > Environment > globalenvironment > View Details > 3 dots > Disable Cluster Auto-Recovery.

Note: In cases where the Aria Suite Lifecycle is patched with Patch 3 or 5 and the vIDM is not patched per KB 412021 then the vIDM cluster gets affected and Remediation via Aria Suite ifecycle will always fail for vIDM even though the vIDM pgpool/postgres cluster issues are addressed and the vIDM cluster is in Healthy state. Do not apply day 2 operations for VIDM in VMware Aria Suite Lifecycle until VIDM is patched.