vSphere HA not getting configured on the hosts with error: "vSphere HA agent cannot be installed or configured"

Products

• VMware vCenter Server

Issue/Introduction

• vSphere HA not getting configured on the hosts
• Error in the UI: "vSphere HA agent cannot be installed or configured"
• In some cases, vCLS VMs will keep  getting deployed and the task gets canceled by user 'com.vmware.vim.eam', and the vCLS VM gets deleted
• 
• If you go to each host, you will find that the vSphere HA state is unhealthy 
• Check for similar logs in the fdm.log following directory /var/log/fdm.log :

  • YYYY-MM-DD HH:MM:SS warning fdm[11432557] [Originator@6876 sub=IO.Connection opID=SWI-72d16df7] Failed to SSL handshake; SSL(<io_obj p:0x0000002a755df438, h:14, <TCP 'X.X.X.X  : 32543'>, <TCP 'X.X.X.X  : 8182'>>), e: 336134278(certificate verify failed), duration: 15msec 
    YYYY-MM-DD HH:MM:SS error fdm[11433613] [Originator@6876 sub=Message opID=SWI-138aae05] Error N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:
    --> PeerThumbprint: 5D:98:54:5A:D8:A7:06:96:F3:8C:F1:88:4A:65:20:0F:4F:D3:F9:53
    --> ExpectedThumbprint: 9A:62:FE:00:4A:0B:BB:13:EE:44:4E:DD:F3:2D:83:49:9C:DE:CE:65
    --> ExpectedPeerName: host-3038
    --> The remote host certificate has these problems:
    -->
    --> * Host name does not match the subject name(s) in certificate.
    -->
    --> * self signed certificate in certificate chain)
    --> [context]zKq7AVECAQAAAERBKQEOZmRtAACoS+ZmZG0AAKMj3gDlB9IA5pTUACua0QA6qdEApkLgACpK4ADJadsADRDbAF1h2wC8+tgBO30AbGlicHRocmVhZC5zby4wAAK9pw5saWJjLnNvLjYA[/context] on handshake 
    YYYY-MM-DD HH:MM:SS error fdm[11432555] [Originator@6876 sub=Message opID=SWI-72d16df7] Error N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:
    --> PeerThumbprint: 5D:98:54:5A:D8:A7:06:96:F3:8C:F1:88:4A:65:20:0F:4F:D3:F9:53
    --> ExpectedThumbprint: 9A:62:FE:00:4A:0B:BB:13:EE:44:4E:DD:F3:2D:83:49:9C:DE:CE:65
    --> ExpectedPeerName: host-3038
    --> The remote host certificate has these problems:
    -->
    --> * Host name does not match the subject name(s) in certificate.
    -->
    --> * self signed certificate in certificate chain)
    --> [context]zKq7AVECAQAAAERBKQEOZmRtAACoS+ZmZG0AAKMj3gDlB9IA5pTUACua0QA6qdEApkLgACpK4ADJadsADRDbAF1h2wC8+tgBO30AbGlicHRocmVhZC5zby4wAAK9pw5saWJjLnNvLjYA[/context] on handshake 
    YYYY-MM-DD HH:MM:SS verbose fdm[11433613] [Originator@6876 sub=Cluster opID=SWI-138aae05] IP X.X.X.X marked bad for reason Unreachable IP 
    YYYY-MM-DD HH:MM:SS info fdm[11433613] [Originator@6876 sub=Message opID=SWI-138aae05] Destroying connection

Environment

VMware vCenter Server 7.0

Resolution

1. Make sure the hosts can ping each other.
2. Disconnect and reconnect the hosts to the vCenter.
   • Please be cautious if the customer has vSAN, NSX, or VDS.
   • N.B. Don't Remove the hosts from the inventory
3. Disable vSphere HA on the cluster
4. Put Cluster in Retreat Mode 
   • https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/7-0/vsphere-resource-management-7-0/creating-a-drs-cluster/vsphere-cluster-services-vcls/putting-a-cluster-in-retreat-mode.html
5. Refresh the certificates on the host as per the following document
   • https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/7-0/vsphere-security-7-0/securing-esxi-hosts/certificate-management-for-esxi-hosts/renew-esxi-certificates.html
6. Re-enable vSphere HA on the cluster
7. Disable Retreat mode on the cluster