vSphere HA not getting configured on the hosts with error: "vSphere HA agent cannot be installed or configured"

Products

• VMware vCenter Server

Issue/Introduction

• vSphere HA not getting configured on the hosts
• Error in the UI: "vSphere HA agent cannot be installed or configured"
• In some cases, vCLS VMs will keep  getting deployed and the task gets canceled by user 'com.vmware.vim.eam', and the vCLS VM gets deleted
• 
• If you go to each host, you will find that the vSphere HA state is unhealthy 
• Check for similar logs in the fdm.log following directory /var/log/fdm.log :

  • YYYY-MM-DD HH:MM:SS warning fdm[11432557] [Originator@6876 sub=IO.Connection opID=SWI-72d16df7] Failed to SSL handshake; SSL(<io_obj p:0x0000002a755df438, h:14, <TCP 'X.X.X.X  : 32543'>, <TCP 'X.X.X.X  : 8182'>>), e: 336134278(certificate verify failed), duration: 15msec 
    YYYY-MM-DD HH:MM:SS error fdm[11433613] [Originator@6876 sub=Message opID=SWI-138aae05] Error N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:
    --> PeerThumbprint: XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX --> ExpectedThumbprint: XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX --> ExpectedPeerName: host-3038 --> The remote host certificate has these problems: --> --> * Host name does not match the subject name(s) in certificate. --> --> * self signed certificate in certificate chain) --> [context]zKq7AVECAQAAAERBKQEOZmRtAACoS+ZmZG0AAKMj3gDlB9IA5pTUACua0QA6qdEApkLgACpK4ADJadsADRDbAF1h2wC8+tgBO30AbGlicHRocmVhZC5zby4wAAK9pw5saWJjLnNvLjYA[/context] on handshake 
    YYYY-MM-DD HH:MM:SS error fdm[11432555] [Originator@6876 sub=Message opID=SWI-72d16df7] Error N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:
    --> PeerThumbprint: XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX --> ExpectedThumbprint: XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX --> ExpectedPeerName: host-3038 --> The remote host certificate has these problems: --> --> * Host name does not match the subject name(s) in certificate. --> --> * self signed certificate in certificate chain) --> [context]zKq7AVECAQAAAERBKQEOZmRtAACoS+ZmZG0AAKMj3gDlB9IA5pTUACua0QA6qdEApkLgACpK4ADJadsADRDbAF1h2wC8+tgBO30AbGlicHRocmVhZC5zby4wAAK9pw5saWJjLnNvLjYA[/context] on handshake 
    YYYY-MM-DD HH:MM:SS verbose fdm[11433613] [Originator@6876 sub=Cluster opID=SWI-138aae05] IP X.X.X.X marked bad for reason Unreachable IP 
    YYYY-MM-DD HH:MM:SS info fdm[11433613] [Originator@6876 sub=Message opID=SWI-138aae05] Destroying connection

Environment

VMware vCenter Server 7.0

Resolution

1. Make sure the hosts can ping each other.
2. Disconnect and reconnect the hosts to the vCenter.
   • Please be cautious if the customer has vSAN, NSX, or VDS.
   • N.B. Don't Remove the hosts from the inventory
3. Disable vSphere HA on the cluster
4. Put Cluster in Retreat Mode 
   • https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/7-0/vsphere-resource-management-7-0/creating-a-drs-cluster/vsphere-cluster-services-vcls/putting-a-cluster-in-retreat-mode.html
5. Refresh the certificates on the host as per the following document
   • https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/7-0/vsphere-security-7-0/securing-esxi-hosts/certificate-management-for-esxi-hosts/renew-esxi-certificates.html
6. Re-enable vSphere HA on the cluster
7. Disable Retreat mode on the cluster

Solution 2 : 

1. Put the host in maintenance mode.

2. Remove the vmware-fdm vib using the below command.

esxcli software vib remove --vibname=vmware-fdm

3. Turn off HA .

4. Exit the host out of maintenance mode. 

5.Disconnect & Reconnect the host.

6. Turn ON vSphere HA.