vCenter HA cluster has lost a node. verify if the passive node with IP address x.x.x.x is up and reachable.

Products

• VMware vCenter Server 7.0
• VMware vCenter Server 8.0
• VMware vCenter Server 6.0

Issue/Introduction

The vcha.log contains below error in the following location "/var/log/vmware/vcha" when SSH'ed to vCenter server. 

YYYY-MM-DDThh:mm:ss error vcha[37400] [Originator@6876 sub=IO.Http opID=SWI-427c3c55] User agent failed to send request; (null), N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception:
Verification parameters:
--> PeerThumbprint: xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:
--> ExpectedThumbprint:
--> ExpectedPeerName: <vCenter FQDN>
--> The remote host certificate has these problems:
-->
--> * unable to get issuer certificate)

The passive node vCenter server vcha logs contains events similar to the one below 

YYYY-MM-DDThh:mm:ss info vcha[41137] [Originator@6876 sub=Cluster opID=SWI-41a7] hostId=xx.xx.xx.xx state=Slave master=xx.xx.xx.xx isolated=false host-list-version=1 config-version=0 vm-metadata-version=0 slv-mst-tdiff-sec=0
YYYY-MM-DDThh:mm:ss error vcha[41143] [Originator@6876 sub=VchaUtil] Error executing command /opt/vmware/vpostgres/current/bin/psql: exit status=[2], stdout=[], stderr=[psql.bin: SSL error: certificate verify failed

Environment

vCenter Server 6.x

vCenter Server 7.x

vCenter Server 8.x

Cause

When vCenter HA is recreated after changes like Machine SSL certificate is replaced with custom certificate. The certificate chain might have missed or any intermediate certificate might have expired or replaced in the customer environment. This results in the certificate not being able to be verified successfully resulting vCenter HA configuration failing.  

Resolution

The solution to fix the issue is to ensure certificate chain is complete and verifiable.

Note: Ensure to take a offline snapshot of vCenter(s) before making changes to the vCenter server.

• Identify the list of certificates needed in the Certificate chain using below command and compare it with the current certificates in the Trusted Store to find the missing certificate.

echo "" | openssl s_client -connect <vCenter FQDN>:443 | openssl x509 -text -noout -fingerprint

• To list all the certificates in the TRUSTED_ROOT, run the below command:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | less

• List the trusted certs published to the VMware Directory Service using the following command.

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert list

• Publish the missing certificate to the TRUSTED_ROOT to complete the chain

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert get --id <CN_id> --login [email protected] --password <PASSWORD> --outcert /tmp/CA_certificate.crt

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert /tmp/CA_certificate.crt

Additional Information

More troubleshooting steps on vCenter HA can be found here