Error: An error occurred while connecting to depot during ESXi vLCM image upgrade in vSphere 7.x and 8.x

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Compliance check or remediation on a cluster using an image-based configuration fails with error:
- An error occurred while connecting to depot or Software Solution com.vmware.nsxt... cannot be found in depot
After enabling Lifecycle Manager and initiating a Compliance Check, vSphere HA fails.
- An error occurred when vCenter Server attempted to initialize the vSphere HA Agent running on the host.
- HA Agent Unreachable - The vSphere HA Agent on the host cannot be reached.
- Cannot complete the configuration of the vSphere HA agent on the host. Applying HA VIBs on the cluster encountered failure.
- A general system error occurred: Installing HA components failed on the cluster: domain-<ID>.
- Cannot find vSphere HA master agent

Performing a vLCM or VUM compliance check on the cluster hangs at 30%, or the task fails with a timeout.

The ESXi host fails to patch with the following log entries in /var/run/log/lifecycle.log:

YYYY-MM-DD-THH:MM:SS lifecycle: 3231338: DepotCollection:331 INFO Could not download from depot at http://<vCenter FQDN>:9084/vum/repository/hostupdate/__micro-depot__vendor-DEL__metadata-233__index__.xml, skipping (('http://<vCenter FQDN>:9084/vum/repository/hostupdate/__micro-depot__vendor-DEL__metadata-233__index__.xml', '', '<urlopen error timed out>'))
YYYY-MM-DD-THH:MM:SS lifecycle: 3231338: DepotMgr:161 ERROR Unable to connect to depot(s) http://<vCenter FQDN>:9084/vum/repository/hostupdate/__micro-depot__vendor-DEL__metadata-233__index__.xml:  ('http://<vCenter FQDN>:9084/vum/repository/hostupdate/__micro-depot__vendor-DEL__metadata-233__index__.xml', '', "Could not download from depot at http://<vCenter FQDN>:9084/vum/repository/hostupdate/__micro-depot__vendor-DEL__metadata-233__index__.xml, skipping (('http://<vCenter FQDN>:9084/vum/repository/hostupdate/__micro-depot__vendor-DEL__metadata-233__index__.xml', '', '<urlopen error timed out>'))") NoneType: None
YYYY-MM-DD-THH:MM:SS lifecycle: 3231338: imagemanagerctl:152 ERROR [DepotConnectError] ([MetadataDownloadError('http://<vCenter FQDN>:9084/vum/repository/hostupdate/__micro-depot__vendor-DEL__metadata-233__index__.xml', '', "Could not download from depot at http://<vCenter FQDN>:9084/vum/repository/hostupdate/__micro-depot__vendor-DEL__metadata-233__index__.xml, skipping (('http://<vCenter FQDN>:9084/vum/repository/hostupdate/__micro-depot__vendor-DEL__metadata-233__index__.xml', '', '<urlopen error timed out>'))")], 'Unable to connect to depot(s) http://<vCenter FQDN>:9084/vum/repository/hostupdate/__micro-depot__vendor-DEL__metadata-233__index__.xml: \n(\'http://<vCenter FQDN>:9084/vum/repository/hostupdate/__micro-depot__vendor-DEL__metadata-233__index__.xml\', \'\', "Could not download from depot at http://<vCenter FQDN>:9084/vum/repository/hostupdate/__micro-depot__vendor-DEL__metadata-233__index__.xml, skipping ((\'http://<vCenter FQDN>:9084/vum/repository/
YYYY-MM-DD-THH:MM:SS lifecycle: hostupdate/__micro-depot__vendor-DEL__metadata-233__index__.xml\', \'\', \'<urlopen error timed out>\'))")')
YYYY-MM-DD-THH:MM:SS lifecycle: 3231338: imagemanagerctl:157 ERROR Traceback (most recent call last):
YYYY-MM-DD-THH:MM:SS lifecycle: 3231338: imagemanagerctl:157 ERROR   File "/usr/lib/vmware/lifecycle/bin/imagemanagerctl.py", line 639, in software
YYYY-MM-DD-THH:MM:SS lifecycle: 3231338: imagemanagerctl:157 ERROR     t.ScanImage(swSpec, opts.depots, opts.taskId)
YYYY-MM-DD-THH:MM:SS lifecycle: 3231338: imagemanagerctl:157 ERROR   File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 1749, in ScanImage
YYYY-MM-DD-THH:MM:SS lifecycle: 3231338: imagemanagerctl:157 ERROR     newPatcherComp, newPatcherVibs = _getPatcherFromImageSpec(imageSpec,
YYYY-MM-DD-THH:MM:SS lifecycle: 3231338: imagemanagerctl:157 ERROR   File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 2484, in _getPatcherFromImageSpec
YYYY-MM-DD-THH:MM:SS lifecycle: 3231338: imagemanagerctl:157 ERROR     specMgr = _getSoftwareSpecMgr(imageSpec, depotUrls)
YYYY-MM-DD-THH:MM:SS lifecycle: 3231338: imagemanagerctl:157 ERROR   File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 2603, in _getSoftwareSpecMgr
YYYY-MM-DD-THH:MM:SS lifecycle: 3231338: imagemanagerctl:157 ERROR     depotMgr = DepotMgr.DepotMgr(depotSpec, connect=True)
YYYY-MM-DD-THH:MM:SS lifecycle: 3231338: imagemanagerctl:157 ERROR   File "/lib64/python3.8/site-packages/vmware/esximage/ImageManager/DepotMgr.py", line 101, in __init__
YYYY-MM-DD-THH:MM:SS lifecycle: 3231338: imagemanagerctl:157 ERROR     self._connect(ignoreError=ignoreError, validate=validate)
YYYY-MM-DD-THH:MM:SS lifecycle: 3231338: imagemanagerctl:157 ERROR   File "/lib64/python3.8/site-packages/vmware/esximage/ImageManager/DepotMgr.py", line 172, in _connect
YYYY-MM-DD-THH:MM:SS lifecycle: 3231338: imagemanagerctl:157 ERROR     _raiseError(depotUrls, errors)
YYYY-MM-DD-THH:MM:SS lifecycle: 3231338: imagemanagerctl:157 ERROR   File "/lib64/python3.8/site-packages/vmware/esximage/ImageManager/DepotMgr.py", line 162, in _raiseError
YYYY-MM-DD-THH:MM:SS lifecycle: 3231338: imagemanagerctl:157 ERROR     raise DepotConnectError(errors, msg)
YYYY-MM-DD-THH:MM:SS lifecycle: 3231338: imagemanagerctl:157 ERROR vmware.esximage.Errors.DepotConnectError: ([MetadataDownloadError('http://<vCenter FQDN>:9084/vum/repository/hostupdate/__micro-depot__vendor-DEL__metadata-233__index__.xml', '', "Could not download from depot at http://<vCenter FQDN>:9084/vum/repository/hostupdate/__micro-depot__vendor-DEL__metadata-233__index

In(14) lifecycle[2112988]: DepotCollection:373 Could not download from depot at https://<vCenter FQDN>:9087/vum/repository/hostupdate/__micro-depot__vendor-vmw__metadata-387__index__.xml, skipping (('https://<vCenter FQDN>:9087/vum/repository/hostupdate/__micro-depot__vendor-vmw__metadata-387__index__.xml', '', '<urlopen error timed out>'))
In(14) lifecycle[2112988]: Downloader:373 Opening https://<vCenter FQDN>:9084/vum/repository/hostupdate/__micro-depot__vendor-DEL__DEL-ESXi-8.0-Addon-cumulative_metadata__index__.xml for download
Wa(12) lifecycle[2112988]: Downloader:210 Download failed: <urlopen error timed out>, 9 retry left...
Wa(12) lifecycle[2112988]: Downloader:210 Download failed: <urlopen error timed out>, 8 retry left...
Wa(12) lifecycle[2112988]: Downloader:210 Download failed: <urlopen error timed out>, 1 retry left...
Er(11) esxupdate[25786092]: An esxupdate error exception was caught:
Er(11) esxupdate[25786092]: Traceback (most recent call last):
Er(11) esxupdate[25786092]: File "/lib64/python3.8/urllib/request.py", line 1354, in do_open
Er(11) esxupdate[25786092]: File "/lib64/python3.8/http/client.py", line 1259, in request
Er(11) esxupdate[25786092]: File "/lib64/python3.8/http/client.py", line 1305, in _send_request
Er(11) esxupdate[25786092]: File "/lib64/python3.8/http/client.py", line 1254, in endheaders
Er(11) esxupdate[25786092]: File "/lib64/python3.8/http/client.py", line 1014, in _send_output
Er(11) esxupdate[25786092]: File "/lib64/python3.8/http/client.py", line 954, in send
Er(11) esxupdate[25786092]: File "/lib64/python3.8/http/client.py", line 1428, in connect
Er(11) esxupdate[25786092]: File "/lib64/python3.8/ssl.py", line 500, in wrap_socket
Er(11) esxupdate[25786092]: File "/lib64/python3.8/ssl.py", line 1073, in _create
Er(11) esxupdate[25786092]: File "/lib64/python3.8/ssl.py", line 1342, in do_handshake
Er(11) esxupdate[25786092]: ConnectionResetError: [Errno 104] Connection reset by peer

Environment

VMware Cloud Foundation (VCF) 4.x, 5.x
VMware vSphere ESXi 7.x, 8.x
VMware vCenter Server 7.x, 8.x

Cause

This issue typically occurs because the ESXi host cannot establish a connection to the vCenter Server over TCP Port 9084.

In vLCM-managed clusters, ESXi hosts must be able to pull software bundles directly from the vCenter software depot over this port. If an intermediate firewall or security group blocks this traffic, the host will be unable to download the required NSX or vendor-specific VIBs.

Resolution

Verification

Log in to the affected ESXi host using SSH.

Verify connectivity to the vCenter Server software depot over TCP port 9084 by running the following command:

nc -z <vCenter-FQDN-or-IP> 9084

Expected output

Success:

Connection to <vCenter-IP> 9084 port [tcp/*] succeeded!

Failure:

nc: connect to <vCenter-IP> port 9084 (tcp) failed: Connection refused

or

nc: connect to <vCenter-IP> port 9084 (tcp) failed: Connection timed out

If the environment is running vSphere 8.0 Update 3 or later, verify connectivity to TCP port 9087, which is used by vSphere Lifecycle Manager to download ESXi updates over HTTPS.
```
nc -z <vCenter-FQDN-or-IP> 9087
```

Remediation

If connectivity verification fails, perform the following steps:

Open the required firewall ports.
Ensure that the following ports are open between the ESXi host and the vCenter Server:
1. TCP port 9084
2. TCP port 9087 (vSphere 8.0 Update 3 and later)
Note: In environments protected by stateful firewalls, communication must be permitted in both directions between the ESXi host and the vCenter Server.
Verify DNS resolution.
Ensure that the ESXi host can resolve the fully qualified domain name (FQDN) of the vCenter Server. If the ESXi host can reach the vCenter Server by IP address but cannot resolve its FQDN, access to the software depot URL will fail.
Retry the lifecycle operation.
After restoring connectivity, return to the vSphere Client and rerun the Check Compliance or Remediate task for the affected cluster.

Additional Information

Check /var/run/log/lifecycle.log to verify which port is being used.
TCP and UDP ports required to access VMware vSphere Update Manager
"An error occurred while connecting to depot" when trying to update the ESXi host using cluster image.