"Download failed: <urlopen error timed out>", Enabling vSphere HA and vLCM Compliance check fails after upgrading vCenter Server to 8.0 U3

Products

• VMware vCenter Server

Issue/Introduction

• After patching vCenter Server to 8.0 U3, HA fails to enable with errors listed below :
  
  
  • An error occurred when vCenter Server attempted to initialize the vSphere HA Agent running on the host.
  • HA Agent Unreachable - The vSphere HA Agent on the host cannot be reached.
  • Cannot complete the configuration of the vSphere HA agent on the host. Applying HA VIBs on the cluster encountered failure.
  • A general system error occurred: Installing HA components failed on the cluster: domain-<ID>.
  • Cannot find vSphere HA master agent
    
    
• Performing vLCM / VUM Compliance check on the Cluster getting hung at 30% or the task fails with timeout
  
  
• vLCM log file on the ESXi host (lifecycle.log) shows timeout errors while connecting to the vLCM depot

/var/run/log/lifecycle.log

In(14) lifecycle[2112988]: DepotCollection:373 Could not download from depot at https://<VC FQDN>:9087/vum/repository/hostupdate/__micro-depot__vendor-vmw__metadata-387__index__.xml, skipping (('h
/vum/repository/hostupdate/__micro-depot__vendor-vmw__metadata-387__index__.xml', '', '<urlopen error timed out>'))
In(14) lifecycle[2112988]: Downloader:373 Opening https://<VC FQDN>:9087/vum/repository/hostupdate/__micro-depot__vendor-DEL__DEL-ESXi-8.0-Addon-cumulative_metadata__index__.xml for download
Wa(12) lifecycle[2112988]: Downloader:210 Download failed: <urlopen error timed out>, 9 retry left...
Wa(12) lifecycle[2112988]: Downloader:210 Download failed: <urlopen error timed out>, 8 retry left...
.
.
Wa(12) lifecycle[2112988]: Downloader:210 Download failed: <urlopen error timed out>, 1 retry left...

Environment

vCenter Server 8.0 U3.

Cause

The Patch Depot URL provided by Update Manager to the hosts (the URL used by the hosts to download the VIBs and Metadata) has been switched to HTTPS (from being HTTP only) in 8.0 U3. This has been done to prevent the security vulnerabilities that are inherent with using a HTTP only connection. This has also resulted in the URL port being switched from 9084 (HTTP) to 9087 (HTTPs).

If there is an external firewall between the vCenter and the ESXi hosts, the port 9087 will have to be explicitly opened to allow the inbound connection to the vCenter from the ESXi hosts. If this is not done, the connection requests will timeout and thus, any relevant operations like host compliance scan, upgrades or enabling HA will fail.

Reference - vCenter Server 8.0 Release Notes

Resolution

This issue is resolved in vCenter Server 8.0 U3a. The fix restores functionality on port 9084.

Workaround

To workaround the issue, open TCP port 9087 on any external firewall that sits between the vCenter and ESXi hosts, to allow inbound connection on port 9087 towards the vCenter.