vSphere HA agent cannot be installed or configured

Products

• VMware vCenter Server

Issue/Introduction

• Upgrade precheck fails with error :
  

  "vSphere HA agent for host <hostname> has an error in <cluster_name> in <workloadomain_name>. vSphere HA agent cannot be installed or configured." 
  

• Cannot configure HA on a cluster and  Re-configuring HA on a cluster failed
  
  
  • Message appears in the vCenter vSphere UI > Host and Cluster view > Cluster > Configure > vSphere Availability

    "The error Cannot install the vCenter Server agent service. "Unknown installer error" 

• In the vCenter /var/log/vmware/vpxd/vpxd.log , following errors will be seen

  YYYY-MM-DDTHH-MM-SS info vpxd[06642] [Originator@6876 sub=Vmomi opID=FdmMonitor-domain-c#-#######] Creating SOAP stub adapter for /fdm on <IP ADDRESS of Host>:443
YYYY-MM-DDTHH-MM-SS info vpxd[06642] [Originator@6876 sub=vmomi.soapStub[54454] opID=FdmMonitor-domain-c#-#######] SOAP request returned HTTP failure; <<io_obj p:0x00007efcd4b372e8, h:78, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-##/fdm>, method: login; code: 503(Service Unavailable); fault: (null)
YYYY-MM-DDTHH-MM-SS warning vpxd[06642] [Originator@6876 sub=Vmomi opID=FdmMonitor-domain-c#-#######] [FdmClientAdapter] FDM service on host <IP ADDRESS of Host> is not ready.
YYYY-MM-DDTHH-MM-SS error vpxd[06642] [Originator@6876 sub=Vmomi opID=FdmMonitor-domain-c#-#######] Failed to login on host; <</hgw/host-##>, /fdm>, N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)

Environment

• vCenter 7.x
• vCenter 8.x

Cause

This issue may result from degraded network connectivity or throughput between the ESXi host and vCenter, which hinders the timely installation of the FDM agent.

Resolution

• Reboot the host as a possible underlying storage issue may be preventing correct access to the datastores on the host.
  
  
• Reconfigure HA. If the problem is not resolved, execute the steps provided below
  
  
• The FDM VIB agent can be installed on the ESXi host(s) by executing the steps outlined below
• •  

• • • Establish an SSH connection to the ESXi host. Confirm the absence of a pre-existing FDM installation as follows:

                         esxcli software vib list | grep fdm

• • • Establish an SSH session to the ESXi host. Confirm reachability of the vCenter by executing a ping command 

                         vmkping -I vmkx destination-ip

• • • Verify that SSH is enabled on the ESXi host and make sure port 22 /443 is not blocked using commands below:  

             nc -z <destination-ip> 22

             openssl s_client -connect <ESXI IP Address>:443  ( Run this command from vCenter )

             openssl s_client -connect <vCenter IP Address>:443  ( Run this command from ESXi Host )

• • • For more information on testing network connectivity - Troubleshooting network and TCP/UDP port connectivity issues on Hosts

• • • Using WinSCP, SCP or a similar protocol, copy the fdm vib file located at /etc/vmware-vpx/docRoot/vSphere-HA-depot/vib20/vmware-fdm/ to the ESXi /tmp folder

• • • • For Example: scp VMware_bootbank_vmware-fdm_6.0.0-3634794.vib root@ESXi_FQDN:/tmp

• • • If the file fails to copy or takes a long time to copy to the host, insufficient network traffic throughput between ESXi and vCenter Server is preventing the agent from installing.

• • • Install the VIB on ESXi :

                         esxcli software vib install -v /tmp/VMware_bootbank_vmware-fdm_#.#.0-########.vib

• • • Confirm that agent is installed by executing : 

                 esxcli software vib list | grep fdm

Additional Information

Impact/Risks:

1. Host cannot not participate in the HA cluster.
2. Cannot proceed with the cluster upgrade.