Pgpool commands are not working in one node of VIDM

Products

• VMware Aria Suite

Issue/Introduction

• PgService and vPostgres are running in the vIDM nodes.
• However, in one of the nodes the pgpool commands from the article Troubleshooting VMware Identity Manager postgres cluster deployed through vRealize Suite Lifecycle Manager request a password and fails with the error ERROR: connection to host "localhost" failed with error "Connection timeout"
• The command su root -c "echo -e 'password'|/usr/local/bin/pcp_watchdog_info -p 9898 -h localhost -U pgpool" will indicate that the status of the node is LOST or SHUTDOWN or DEAD. 

Environment

• VMware Identity Manager 3.3.x

Cause

This can happens due to network outage (bug in pgpool service)

Resolution

1. Validate if the Postgres Cluster service and vPostgres are running in the unhealthy node. 
   

   /etc/init.d/pgService status

   /etc/init.d/vpostgres status

    

2. If required start vPostgres or Postgres Cluster service(s) and then retry the pgpool commands from the article  Troubleshooting VMware Identity Manager postgres cluster deployed through vRealize Suite Lifecycle Manager
   
   

   /etc/init.d/pgService start
   
   /etc/init.d/vpostgres start

   
   

3. Try to gracefully stop and start the PgService.
   

   /etc/init.d/pgService stop
   
   /etc/init.d/vpostgres stop
   
   rm -rf /tmp/pgpool_status
   
   /etc/init.d/vpostgres start
   
   /etc/init.d/pgService start

    

4. IF pgService Not stopping on Nodes or DEAD / SHUTDOWN Nodes:
   

   Run below command for force stop
   
   pkill -e -9 pgpool 
   rm /tmp/.s.PGSQL.* 
   rm /tmp/.s.PGPOOL.* 
   rm /tmp/pgpool_status 
   rm /var/run/pgpool/pgpool.pid 
   rm -rf /var/run/pgpool/socket
   fuser -k 9999/tcp
   
   /etc/init.d/pgService status
   /etc/init.d/pgService start

    

5. If pgService service was stuck while start pgService
   
   Forcefully restarted the pgService on both the DEAD / SHUTDOWN nodes

    /etc/init.d/pgService stop
   
    fuser -k 9999/tcp >/dev/null 2>&1 
     rm /tmp/.s.PGSQL.* >/dev/null 2>&1
     rm /tmp/.s.PGPOOL.* >/dev/null 2>&1
     rm  /tmp/pgpool_status >/dev/null 2>$1
     rm /var/run/pgpool/pgpool.pid >/dev/null 2>&1
     rm -rf /var/run/pgpool/socket >/dev/null 2>&1
     
    /etc/init.d/pgService status
    /etc/init.d/pgService start
   

    

6. If the node goes QUARANTINE run below command on primary node
   

   su root -c "cat /usr/local/etc/pgpool.pwd|/usr/local/bin/pcp_recovery_node -v -h delegateIP -p 9898 -U pgpool -n <node_id>"