vCenter HA configuration is failing with error message "PostgreSQL replication is not in progress. Verify if PostgreSQL server is running on the Passive node and that the Passive node is reachable on the vCenter HA network"

Products

• VMware vCenter Server

Issue/Introduction

• While configuring vCenter HA below error is encountered:
  
  
  
• This may also lead to passive node going down as seen below:
  

• The vmware-vpostgres service may fail to start after executing the "vcha-destroy -f" command, to remove the VCHA configuration from the vCenter Server.
  
  
• Logs at /var/log/vmware/vpostgres/serverlog.stderr showed the following:

FATAL: could not load server certificate file "/storage/db/vpostgres_ssl/server.crt": SSL Error code 2147483650

or 

FATAL: could not load server certificate file "/storage/db/vpostgres_ssl/server.crt": No such file or directory


• When reviewing the passive node's /var/log/vmware/vcha/vcha.log, the below errors are observed: 
  
  YYYY-MM-DDTHH:MM:SSZ info vcha[21360] [Originator@6876 sub=Cluster opID=###-####] hostId=##.##.##.## state=Slave master=##.##.##.## isolated=false host-list-version=1 config-version=0 vm-metadata-version=0 slv-mst-tdiff-sec=0
YYYY-MM-DDTHH:MM:SSZ error vcha[21381] [Originator@6876 sub=VchaUtil] Error executing command /opt/vmware/vpostgres/current/bin/psql: exit status=[2], stdout=[], stderr=[psql.bin: error: SSL error: certificate verify failed
YYYY-MM-DDTHH:MM:SSZ error vcha[21381] [Originator@6876 sub=VchaUtil] Error executing command /opt/vmware/vpostgres/current/bin/psql: exit status=[2], stdout=[], stderr=[psql.bin: error: SSL error: certificate verify failed

Environment

VMware vCenter Server 8.x
VMware vCenter Server 7.x

Cause

This was caused due to a mismatch between the vPostgres SSL certificate and the SSL certificate of the vCenter.

Resolution

To update the certificate and resolve the issue, follow these steps:

1. Destroy the VCHA configuration.
   
   
2. Back up the current machine SSL certificate and it's key:
   /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /var/core/Machine_SSL.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /var/core/Machine_SSL.key


3. Navigate to /storage/db/vpostgres_ssl and back up the existing server certificate and key:
   cd /storage/db/vpostgres_ssl
cp server.crt server.crt.bkp
cp server.key server.key.bkp
   
   
4. Replace contents in server.crt with /var/core/Machine_SSL.crt:
   cp /var/core/Machine_SSL.crt /storage/db/vpostgres_ssl/server.crt


5. Replace contents in server.key with /var/core/Machine_SSL.key:
   cp /var/core/Machine_SSL.key /storage/db/vpostgres_ssl/server.key


6. Navigate to /storage/db/vpostgres_ssl/ location to have the required permissions:
   chmod 0600 server.crt server.key
chown vpostgres:vpgmongrp server.crt server.key


   
   
7. Restart all services:
   service-control --stop --all && service-control --start --all


8. Reconfigure the VCHA.