vSphere HA agent cannot be installed or configured

Products

• VMware vCenter Server

Issue/Introduction

• Upgrade precheck fails with error :
  

  "vSphere HA agent for host <hostname> has an error in <cluster_name> in <workloadomain_name>. vSphere HA agent cannot be installed or configured." 
  

• Cannot configure HA on a cluster and  Re-configuring HA on a cluster failed
  
  
  • Message appears in the vCenter vSphere UI > Host and Cluster view > Cluster > Configure > vSphere Availability

    "The error Cannot install the vCenter Server agent service. "Unknown installer error" 

• In the vCenter /var/log/vmware/vpxd/vpxd.log , following errors will be seen

  YYYY-MM-DDTHH-MM-SS info vpxd[06642] [Originator@6876 sub=Vmomi opID=FdmMonitor-domain-c7-7d252855] Creating SOAP stub adapter for /fdm on <IP ADDRESS of Host>:443
YYYY-MM-DDTHH-MM-SS info vpxd[06642] [Originator@6876 sub=vmomi.soapStub[54454] opID=FdmMonitor-domain-c7-7d252855] SOAP request returned HTTP failure; <<io_obj p:0x00007efcd4b372e8, h:78, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-10/fdm>, method: login; code: 503(Service Unavailable); fault: (null)
YYYY-MM-DDTHH-MM-SS warning vpxd[06642] [Originator@6876 sub=Vmomi opID=FdmMonitor-domain-c7-7d252855] [FdmClientAdapter] FDM service on host <IP ADDRESS of Host> is not ready.
YYYY-MM-DDTHH-MM-SS error vpxd[06642] [Originator@6876 sub=Vmomi opID=FdmMonitor-domain-c7-7d252855] Failed to login on host; <</hgw/host-10>, /fdm>, N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)

Environment

vCenter 7.x

vCenter 8.x

Cause

This issue may result from degraded network connectivity or throughput between the ESXi host and vCenter, which hinders the timely installation of the FDM agent.

Resolution

Reconfigure HA. If the problem is not resolved, execute the steps provided below

The FDM VIB agent can be installed on the ESXi host(s) by executing the steps outlined below

• • • Establish an SSH connection to the ESXi host. Confirm the absence of a pre-existing FDM installation as follows:

esxcli software vib list |grep fdm

• • • Establish an SSH connection to the ESXi host. Confirm the absence of a pre-existing FDM installation as follows:

                         esxcli software vib list |grep fdm

• • • Establish an SSH session to the ESXi host. Confirm reachability of the vCenter by executing a ping command 

                         vmkping -I vmkx destination-ip

• • • Verify that SSH is enabled on the ESXi host and make sure port 22 /443 is not blocked using commands below:  

             nc -z <destination-ip> 22

             openssl s_client -connect <ESXI IP Address>:443  ( Run this command from vCenter )

             openssl s_client -connect <vCenter IP Address>:443  ( Run this command from ESXi Host )

• • • For more information on testing network connectivity - Troubleshooting network and TCP/UDP port connectivity issues on Hosts

• • • Using WinSCP, SCP or a similar protocol, copy the fdm vib file located at /etc/vmware-vpx/docRoot/vSphere-HA-depot/vib20/vmware-fdm/ to the ESXi /tmp folder,

• • • For Example: scp VMware_bootbank_vmware-fdm_6.0.0-3634794.vib root@ESXi_FQDN:/tmp

• • • If the file fails to copy or takes a long time to copy to the host, insufficient network traffic throughput between ESXi and vCenter Server is preventing the agent from installing.

• • • Install the VIB on ESXi :

                         esxcli software vib install -v /tmp/VMware_bootbank_vmware-fdm_x.x.0-xxxxxx.vib

• • • Confirm that agent is installed by executing : 

                 esxcli software vib list |grep fdm

Additional Information

Impact/Risks:

1. Host cannot not participate in the HA cluster.
2. Cannot proceed with the cluster upgrade.