Enabling vSphere HA and vLCM Compliance check fails after upgrading vCenter Server to 8.0 U3

Products

• VMware vCenter Server
• VMware vCenter Server 8.0

Issue/Introduction

• After patching vCenter Server to 8.0 U3, HA fails to enable with errors listed below :
  
  
  • An error occurred when vCenter Server attempted to initialize the vSphere HA Agent running on the host.
  • HA Agent Unreachable - The vSphere HA Agent on the host cannot be reached.
  • Cannot complete the configuration of the vSphere HA agent on the host. Applying HA VIBs on the cluster encountered failure.
  • A general system error occurred: Installing HA components failed on the cluster: domain-<ID>.
  • Cannot find vSphere HA master agent
    
    
• Performing vLCM / VUM Compliance check on the Cluster getting hung at 30% or the task fails with timeout
  
  
• vLCM log file on the ESXi host (lifecycle.log) shows timeout errors while connecting to the vLCM depot

/var/run/log/lifecycle.log

In(14) lifecycle[2112988]: DepotCollection:373 Could not download from depot at https://<VC FQDN>:9087/vum/repository/hostupdate/__micro-depot__vendor-vmw__metadata-387__index__.xml, skipping (('h
/vum/repository/hostupdate/__micro-depot__vendor-vmw__metadata-387__index__.xml', '', '<urlopen error timed out>'))
In(14) lifecycle[2112988]: Downloader:373 Opening https://<VC FQDN>:9087/vum/repository/hostupdate/__micro-depot__vendor-DEL__DEL-ESXi-8.0-Addon-cumulative_metadata__index__.xml for download
Wa(12) lifecycle[2112988]: Downloader:210 Download failed: <urlopen error timed out>, 9 retry left...
Wa(12) lifecycle[2112988]: Downloader:210 Download failed: <urlopen error timed out>, 8 retry left...
.
.
Wa(12) lifecycle[2112988]: Downloader:210 Download failed: <urlopen error timed out>, 1 retry left...
.
.
Er(11) esxupdate[25786092]: An esxupdate error exception was caught:
Er(11) esxupdate[25786092]: Traceback (most recent call last):
Er(11) esxupdate[25786092]: File "/lib64/python3.8/urllib/request.py", line 1354, in do_open
Er(11) esxupdate[25786092]: File "/lib64/python3.8/http/client.py", line 1259, in request
Er(11) esxupdate[25786092]: File "/lib64/python3.8/http/client.py", line 1305, in _send_request
Er(11) esxupdate[25786092]: File "/lib64/python3.8/http/client.py", line 1254, in endheaders
Er(11) esxupdate[25786092]: File "/lib64/python3.8/http/client.py", line 1014, in _send_output
Er(11) esxupdate[25786092]: File "/lib64/python3.8/http/client.py", line 954, in send
Er(11) esxupdate[25786092]: File "/lib64/python3.8/http/client.py", line 1428, in connect
Er(11) esxupdate[25786092]: File "/lib64/python3.8/ssl.py", line 500, in wrap_socket
Er(11) esxupdate[25786092]: File "/lib64/python3.8/ssl.py", line 1073, in _create
Er(11) esxupdate[25786092]: File "/lib64/python3.8/ssl.py", line 1342, in do_handshake
Er(11) esxupdate[25786092]: ConnectionResetError: [Errno 104] Connection reset by peer

Environment

vCenter Server 8.0 U3

Cause

• The Patch Depot URL provided by Update Manager to the hosts (the URL used by the hosts to download the VIBs and Metadata) has been switched to HTTPS (from being HTTP only) in 8.0 U3. This has been done to prevent the security vulnerabilities that are inherent with using a HTTP only connection. This has also resulted in the URL port being switched from 9084 (HTTP) to 9087 (HTTPs).
• If there is an external firewall between the vCenter and the ESXi hosts, the port 9087 will have to be explicitly opened to allow the inbound connection to the vCenter from the ESXi hosts. If this is not done, the connection requests will timeout and thus, any relevant operations like host compliance scan, upgrades or enabling HA will fail.

Reference: VMware vCenter Server 8.0 Update 3 Release Notes

Resolution

• This issue is resolved in vCenter Server 8.0 U3a and later. The fix restores functionality on port 9084.

Workaround:

• To workaround the issue, open TCP port 9087 on any external firewall that sits between the vCenter and ESXi hosts, to allow inbound connection on port 9087 towards the vCenter.

Additional Information

To test the connectivity between ESXi and vCenter Server. Open an SSH session to one of the impacted ESXi hosts and run the below command

nc -z vCenter_IP 9087

Expected response is:

Connection to vCenter_IP 9087 port [tcp/*] succeeded!