vSphere HA agent cannot be installed or configured

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• Upgrade precheck fails with error :
  
  "vSphere HA agent for host <hostname> has an error in <cluster_name> in <workloadomain_name>. vSphere HA agent cannot be installed or configured." 
  
  
• Cannot configure HA on a cluster and  Re-configuring HA on a cluster failed
  
  
  • Message appears in the vCenter vSphere UI > Host and Cluster view > Cluster > Configure > vSphere Availability
    
    "The error Cannot install the vCenter Server agent service. "Unknown installer error" 

• In the vCenter /var/log/vmware/vpxd/vpxd.log , following errors will be seen

  YYYY-MM-DDTHH-MM-SS info vpxd[06642] [Originator@6876 sub=Vmomi opID=FdmMonitor-domain-c7-7d252855] Creating SOAP stub adapter for /fdm on <IP ADDRESS of Host>:443
YYYY-MM-DDTHH-MM-SS info vpxd[06642] [Originator@6876 sub=vmomi.soapStub[54454] opID=FdmMonitor-domain-c7-7d252855] SOAP request returned HTTP failure; <<io_obj p:0x00007efcd4b372e8, h:78, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-10/fdm>, method: login; code: 503(Service Unavailable); fault: (null)
YYYY-MM-DDTHH-MM-SS warning vpxd[06642] [Originator@6876 sub=Vmomi opID=FdmMonitor-domain-c7-7d252855] [FdmClientAdapter] FDM service on host <IP ADDRESS of Host> is not ready.
YYYY-MM-DDTHH-MM-SS error vpxd[06642] [Originator@6876 sub=Vmomi opID=FdmMonitor-domain-c7-7d252855] Failed to login on host; <</hgw/host-10>, /fdm>, N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)

Environment

• VMware vCenter Server 7.0
• VMware vCenter Server 8.0
• VMware vSphere ESXi 7.0
• VMware vSphere ESXi 8.0

Cause

This issue can occurs when network traffic throughput between the ESXi and vCenter Server is insufficient and the vCenter Server cannot push the FDM agent to the ESX host over the network in a timely manner.

Resolution

Workaround: 

Reconfigure HA if the workaround does not work, then follow the below steps.

 

Run through the steps below to install the FDM VIB agent on the ESXi host(s)

• • • Connect to the ESXi with SSH
       
      • Confirm that fdm isn't installed already :
        • esxcli software vib list |grep fdm
          
          
    • Connect to the vCenter Server Appliance using SSH.
      
      
    • Verify that you can ping the ESX host: 
      • vmkping -I vmkx destination-ip
        
        
    • Verify that SSH is enabled on the ESXi host and that port 22 /443 is not blocked:
        
      • nc -z <destination-ip> 22
      • openssl s_client -connect <ESXI IP Address>:443  ( Run this command from vCenter )
      • openssl s_client -connect <vCenter IP Address>:443  ( Run this command from ESXi Host )
    • For more information on testing network connectivity see here
      
      
    • Using WinSCP, SCP or a similar protocol, copy the fdm vib file located at /etc/vmware-vpx/docRoot/vSphere-HA-depot/vib20/vmware-fdm/ to the ESXi /tmp folder,
       
      • For Example: scp VMware_bootbank_vmware-fdm_6.0.0-3634794.vib root@ESXi_FQDN:/tmp
        
        
    • If the file fails to copy or takes a long time to copy to the host, insufficient network traffic throughput between ESXi and vCenter Server is preventing the agent from installing.
      
      
    • Install the VIB on ESXi :
      • esxcli software vib install -v /tmp/VMware_bootbank_vmware-fdm_x.x.0-xxxxxx.vib
        
        
    • Confirm that agent is installed by running : 
      • esxcli software vib list |grep fdm

Additional Information

Impact/Risks:

1. Host cannot not participate in the HA cluster.
2. Cannot proceed with the cluster upgrade.

 

-----------------------------------------------------------

 

The same steps bellow can be used to resolve issues with the HA vib's that is stopping hosts in a HA cluster completing election or stopping hosts communicating with the HA agent

 

In cases like this, you will see error's like this in vCenter