Error: "vSphere HA agent cannot be installed or configured" due to degraded network connectivity

Products

• VMware vCenter Server

Issue/Introduction

• Upgrade precheck fails with error:
  

  vSphere HA agent for host <hostname> has an error in <cluster_name> in <workloadomain_name>. vSphere HA agent cannot be installed or configured.
  

• Both configuring HA on a cluster and re-configuring HA on a cluster fails
  
  
• Message appears in the vCenter vSphere UI > Host and Cluster view > Cluster > Configure > vSphere Availability

  The error Cannot install the vCenter Server agent service. "Unknown installer error

• In the vCenter /var/log/vmware/vpxd/vpxd.log, the following errors can be seen:

  YYYY-MM-DDTHH-MM-SS info vpxd[06642] [Originator@6876 sub=Vmomi opID=FdmMonitor-domain-c#-#######] Creating SOAP stub adapter for /fdm on <IP ADDRESS of Host>:443
YYYY-MM-DDTHH-MM-SS info vpxd[06642] [Originator@6876 sub=vmomi.soapStub[54454] opID=FdmMonitor-domain-c#-#######] SOAP request returned HTTP failure; <<io_obj p:0x00007efcd4b372e8, h:78, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-##/fdm>, method: login; code: 503(Service Unavailable); fault: (null)
YYYY-MM-DDTHH-MM-SS warning vpxd[06642] [Originator@6876 sub=Vmomi opID=FdmMonitor-domain-c#-#######] [FdmClientAdapter] FDM service on host <IP ADDRESS of Host> is not ready.
YYYY-MM-DDTHH-MM-SS error vpxd[06642] [Originator@6876 sub=Vmomi opID=FdmMonitor-domain-c#-#######] Failed to login on host; <</hgw/host-##>, /fdm>, N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)

Environment

• vCenter 7.x
• vCenter 8.x

Cause

This issue may result from degraded network connectivity or throughput between the ESXi host and vCenter, which hinders the timely installation of the FDM agent.

Resolution

1. Reboot the host, as a possible underlying storage issue may be preventing correct access to the datastores on the host.
   
   
2. Reconfigure HA. If the problem is not resolved, execute the steps provided below.
   
   
3. The FDM VIB agent can be installed on the ESXi host(s) by executing the steps outlined below:
   1. Establish an SSH connection to the ESXi host. Confirm the absence of a pre-existing FDM installation as follows:
      
      esxcli software vib list | grep fdm
      
      
   2. Confirm reachability of the vCenter by executing a ping command:
      
      vmkping -I vmkx destination-ip
      
      
   3. Verify that SSH is enabled on the ESXi host and make sure port 22 / 443 is not blocked:
      
      • On vCenter, run:
        
        openssl s_client -connect <ESXI IP Address>:443
        
        
      • On ESXi, run:
        
        nc -z <destination-ip> 22

openssl s_client -connect <vCenter IP Address>:443
        
        Note: For more information on testing network connectivity, refer to KB Troubleshooting network and TCP/UDP port connectivity issues on Hosts.
        
        
   4. Using WinSCP, SCP or a similar protocol, copy the fdm vib file located on vCenter - /etc/vmware-vpx/docRoot/vSphere-HA-depot/vib20/vmware-fdm/ to the ESXi /tmp folder.

      Example

      1. SSH to vCenter

      2. Copy the fdm vib file from vCenter to ESXi:

         scp /etc/vmware-vpx/docRoot/vSphere-HA-depot/vib20/vmware-fdm/VMware_bootbank_vmware-fdm_6.0.0-3634794.vib root@<ESXi_FQDN>:/tmp

   5. If the file fails to copy or takes a long time to copy to the host, there is insufficient network traffic throughput between ESXi and vCenter Server and is preventing the agent from installing.

• • • Install the VIB on ESXi:
      
      esxcli software vib install -v /tmp/VMware_bootbank_vmware-fdm_#.#.0-########.vib

• • • Confirm that agent is installed:
      
      esxcli software vib list | grep fdm