vSphere HA Agent Fails — "Applying HA VIBs on the cluster encountered a failure"

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

While configuring vSphere HA in the cluster following errors can be seen

HA Agent Unreachable — The vSphere HA Agent on the host cannot be reached.
Cannot complete the configuration of the vSphere HA agent on the host. Applying HA VIBs on the cluster encountered failure.
A general system error occurred: Installing HA components failed on the cluster: domain-<ID>.
vSphere HA configure task fails with:

Cannot complete the configuration of the vSphere HA agent on the host. "Applying HA VIBs on the cluster encountered a failure". Failed installing HA component on the host: host-XXX.
Performing a vLCM / VUM Compliance check on the cluster hangs at 30% or fails with a timeout.

From /var/run/log/lifecycle.log on the ESXi host:

YYYY-MM-DDThh:ss:msZ In(14) lifecycle[20373705]: imagemanagerctl:843 Calling with arguments: components --apply --task-id 5256cdb1-XXXX-cc22-XXXX-7713XXXX37e6b --depot http://<FQDN_of_vCenter>:9084/vum/repository/hostupdate/__micro-depot__vendor-vmw__vmw-ESXi-8.0.3-metadata__index__.xml --depot http://<FQDN_of_vCenter>:9084/vum/repository/hostupdate/__micro-depot__vendor-NTA__metadata-177
__index__.xml --depot http://FQDN_of_vCenter>:9084/vum/repository/hostupdate/__micro-depot__vendor-vmw__metadata-167__index__.xml --depot http://<FQDN_of_vCenter>:9084/vum/repository/hostupdate/__micro-depot__vendor-vmw__vmw-ESXi-8.0-vmtools-12.5-metadata__index__.xml --component vsphere-fdm:8.0.3-24674346
YYYY-MM-DDThh:ss:msZWa(12) lifecycle[20373705]: Downloader:214 Download failed: <urlopen error timed out>, 9 retry left...
YYYY-MM-DDThh:ss:msWa(12) lifecycle[20373705]: Downloader:214 Download failed: <urlopen error timed out>, 8 retry left...
YYYY-MM-DDThh:ss:msZ Wa(12) lifecycle[20373705]: Downloader:214 Download failed: <urlopen error timed out>, 7 retry left...

From /var/log/vmware/vpxd/vpxd.log on the vCenter Server:

YYYY-MM-DDThh:mm:ss warning vpxd[06963] [Originator@6876 sub=Vmomi opID=FdmMonitor-domain-c80094-XXXXX] Got vmacore exception when invoking VMOMI method; <</hgw/host-81084>, /fdm>, csi.FdmService.GetAbout, N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)
--> [context]zKq7AVECAQAAACqAeAEbdnB4ZAAAQxxTbGlidm1hY29yZS5zbwAACBhCACk/QwCWmUoBIEIebGlidm1vbWkuc28AAT9kIQHqkCEBXwohggvnSwF2cHhkAILbL0wBguM7TAGC4DxMAYLv1UsBgucDTAEB19oaA9/GBGxpYmNzaS10eXBlcy5zbwCCfEw/AYK6jqgBgoqTqAGC71B+AoKuUX4CAAdROAAE7DcAF0U4AMUPUQSwjgBsaWJwdGhyZWFkLnNvLjAABd/6D2xpYmMuc28uNgA=[/context]
YYYY-MM-DDThh:mm:ss error vpxd[06963] [Originator@6876 sub=HostUpgrader opID=FdmMonitor-domain-c80094-3e29ba48] Failed to get fdm aboutInfo from host-XXX: N5Vmomi5Fault17HostCommunication9ExceptionE(Fault cause: vmodl.fault.HostCommunication)

Cause

The vCenter Update Manager port (updateManager) is blocked on all ESXi hosts in the cluster, preventing communication needed for HA component installation.

Resolution

Steps to check if the Update Manager port is blocked:

SSH into the ESXi host.
Run the following command:
esxcli network firewall ruleset list

Check the output for updateManager. If you see: updateManager false then the port is currently blocked.

esxcli network firewall ruleset list
Name                         Enabled  Enable/Disable configurable  Allowed IP configurable
---------------------------  -------  ---------------------------  -----------------------
CIMHttpServer                  false                        false                     true
CIMHttpsServer                 false                        false                     true
CIMSLP                         false                        false                     true
iSCSI                          false                        false                     true
vpxHeartbeats                   true                        false                     true
updateManager                  false                         true                     true
faultTolerance                  true                         true                     true

How to enable the Update Manager via CLI:

SSH into the ESXi host.
Run this command:

esxcli network firewall ruleset set -e true -r updateManager

Verify the change:

esxcli network firewall ruleset list

esxcli network firewall ruleset list

Name                         Enabled  Enable/Disable configurable  Allowed IP configurable
---------------------------  -------  ---------------------------  -----------------------
CIMHttpServer                  false                        false                     true
CIMHttpsServer                 false                        false                     true
CIMSLP                         false                        false                     true
iSCSI                          false                        false                     true
vpxHeartbeats                  true                         false                     true
updateManager                  true                         true                      true
faultTolerance                 true                         true                      true

How to open the firewall in the ESXi host UI:

In the vSphere Client, go to the ESXi host.
Navigate to Configure → Firewall → Edit.
Search for vCenter Update Manager.
Enable (check) the port.
Click OK to apply.

Additional Information

Validate traffic is routable from the vCenter to the ESXi host over TCP ports 80, 443, 902, and 9080 using the following command :

curl -v telnet://<ESXi-Host-IP>:<Port-Number>
Validate traffic is routable from the ESXi host to the vCenter over TCP ports 9084 and 9087 using the following command :

nc -z <VCENTER-IP> <Port-Number>
There should also be bidirectional communication between all hosts in the cluster on 8182 TCP and UDP, validate using following command :

nc -z <ESXi-Host-IP> <Port-Number>
Refer following document for port requirements VMware Ports and Protocols