After enabling Lifecycle Manager and initiating a Compliance Check, vSphere HA fails

Products

• VMware vCenter Server

Issue/Introduction

• After setting up Lifecycle Manager and initiating a Compliance Check, vSphere HA fails with the following errors: 
  
  
• • An error occurred when vCenter Server attempted to initialize the vSphere HA Agent running on the host.
  • HA Agent Unreachable - The vSphere HA Agent on the host cannot be reached.
  • Cannot complete the configuration of the vSphere HA agent on the host. Applying HA VIBs on the cluster encountered failure.
  • A general system error occurred: Installing HA components failed on the cluster: domain-<ID>.
  • Cannot find vSphere HA master agent
    
    
• Performing vLCM / VUM Compliance check on the Cluster getting hung at 30% or the task fails with timeout
  
  
• vLCM log file on the ESXi host (lifecycle.log) shows timeout errors while connecting to the vLCM depot

/var/run/log/lifecycle.log

In(14) lifecycle[2112988]: DepotCollection:373 Could not download from depot at https://<VC FQDN>:9087/vum/repository/hostupdate/__micro-depot__vendor-vmw__metadata-387__index__.xml, skipping (('h
/vum/repository/hostupdate/__micro-depot__vendor-vmw__metadata-387__index__.xml', '', '<urlopen error timed out>'))
In(14) lifecycle[2112988]: Downloader:373 Opening https://<VC FQDN>:9084/vum/repository/hostupdate/__micro-depot__vendor-DEL__DEL-ESXi-8.0-Addon-cumulative_metadata__index__.xml for download
Wa(12) lifecycle[2112988]: Downloader:210 Download failed: <urlopen error timed out>, 9 retry left...
Wa(12) lifecycle[2112988]: Downloader:210 Download failed: <urlopen error timed out>, 8 retry left...
.
.
Wa(12) lifecycle[2112988]: Downloader:210 Download failed: <urlopen error timed out>, 1 retry left...
.
.
Er(11) esxupdate[25786092]: An esxupdate error exception was caught:
Er(11) esxupdate[25786092]: Traceback (most recent call last):
Er(11) esxupdate[25786092]: File "/lib64/python3.8/urllib/request.py", line 1354, in do_open
Er(11) esxupdate[25786092]: File "/lib64/python3.8/http/client.py", line 1259, in request
Er(11) esxupdate[25786092]: File "/lib64/python3.8/http/client.py", line 1305, in _send_request
Er(11) esxupdate[25786092]: File "/lib64/python3.8/http/client.py", line 1254, in endheaders
Er(11) esxupdate[25786092]: File "/lib64/python3.8/http/client.py", line 1014, in _send_output
Er(11) esxupdate[25786092]: File "/lib64/python3.8/http/client.py", line 954, in send
Er(11) esxupdate[25786092]: File "/lib64/python3.8/http/client.py", line 1428, in connect
Er(11) esxupdate[25786092]: File "/lib64/python3.8/ssl.py", line 500, in wrap_socket
Er(11) esxupdate[25786092]: File "/lib64/python3.8/ssl.py", line 1073, in _create
Er(11) esxupdate[25786092]: File "/lib64/python3.8/ssl.py", line 1342, in do_handshake
Er(11) esxupdate[25786092]: ConnectionResetError: [Errno 104] Connection reset by peer

Environment

vCenter Server 8.x

Cause

Communication over port 9084 is unsuccessful. 

Ensure external firewalls between vCenter and ESXi hosts allow inbound connection to vCenter from the ESXi hosts over port 9084. If the firewall does not have port 9084 open, tasks such as compliance scans, upgrades, or enabling vSphere HA will fail. 

Resolution

Open TCP port 9084 on any external firewall that sits between the vCenter and ESXi hosts.

 

To test the connectivity between ESXi and vCenter Server. SSH to an ESXi host and run this command: 

nc -z vCenter_IP 9084

Expected response is:

Connection to vCenter_IP 9084 port [tcp/*] succeeded!

If nothing is returned, the connection has failed. 

Additional Information

Starting from 8.0 Update 3, the vSphere Lifecycle Manager downloads updates for ESXi hosts by a HTTPS connection to the vCenter instance on port 9087. Port 9087 may also need to be opened.

Check /var/run/log/lifecycle.log to verify which port is being used.