vCenter HA cluster has lost a node. Verify if the passive node with IP address <IP Address> is up and reachable.

Products

• VMware vCenter Server 7.0
• VMware vCenter Server 8.0
• VMware vCenter Server 6.0

Issue/Introduction

• The vCenter /var/log/vmware/vcha/vcha.log contains below error.

YYYY-MM-DDThh:mm:ss error vcha[37400] [Originator@6876 sub=IO.Http opID=SWI-########] User agent failed to send request; (null), N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception:
Verification parameters:
--> PeerThumbprint: ##:##:##:##:##:##:##:##:##:##:##:##:##:##:##
--> ExpectedThumbprint:
--> ExpectedPeerName: <vCenter FQDN>
--> The remote host certificate has these problems:
-->
--> * unable to get issuer certificate)


• The passive node vCenter server  /var/log/vmware/vcha/vcha.log contains events similar to the one below 

YYYY-MM-DDThh:mm:ss info vcha[41137] [Originator@6876 sub=Cluster opID=SWI-41a7] hostId=##:##:##:## state=Slave master=##:##:##:## isolated=false host-list-version=1 config-version=0 vm-metadata-version=0 slv-mst-tdiff-sec=0
YYYY-MM-DDThh:mm:ss error vcha[41143] [Originator@6876 sub=VchaUtil] Error executing command /opt/vmware/vpostgres/current/bin/psql: exit status=[2], stdout=[], stderr=[psql.bin: SSL error: certificate verify failed

Environment

• vCenter Server 6.x
• vCenter Server 7.x
• vCenter Server 8.x

Cause

When vCenter HA is recreated after changes like Machine SSL certificate replacement, the certificate chain might have missed or any intermediate certificate might have been expired or replaced in the environment.

Resolution

The solution to fix the issue is to ensure certificate chain is complete and verifiable.

Note: Ensure to take an offline snapshot of vCenter(s) before making changes to the vCenter server. Refer: Snapshot Best practices for vCenter Server Virtual Machines

1. Identify the list of certificates needed in the Certificate chain using below command and compare it with the current certificates in the Trusted Store to find the missing certificate.
   
   echo "" | openssl s_client -connect <vCenter FQDN>:443 | openssl x509 -text -noout -fingerprint
   
   
2. To list all the certificates in the TRUSTED_ROOT, run the below command:
   
   /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | less
   
   
3. List the trusted certs published to the VMware Directory Service using the following command.
   
   /usr/lib/vmware-vmafd/bin/dir-cli trustedcert list
   
   
4. Publish the missing certificate to the TRUSTED_ROOT to complete the chain
   
   /usr/lib/vmware-vmafd/bin/dir-cli trustedcert get --id <CN_id> --login [email protected] --password <PASSWORD> --outcert /tmp/CA_certificate.crt

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert /tmp/CA_certificate.crt

Additional Information

More troubleshooting steps on vCenter HA can be found here