Host upgrade fails in low bandwidth Remote Office and Branch Office (ROBO) setup

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

This article explains potential failures in ROBO environments with limited network bandwidth and provides possible solutions.

Symptoms:

If the hosts are in a ROBO (Remote Office/Branch Office) setup and have low bandwidth (< 5 Mbps), upgrading these hosts using VUM or vLCM results in a failure. The errors that can be seen are listed below:

 

• During the vLCM upgrade:
  An error occurred while downloading VIB from the URL http://<VC>:9084/vum/repository/hostupdate/vmw/vib20/loadesx/<vibname>.vib
  
  
• During the VUM upgrade:
  The host cannot download files from the VMware vSphere Lifecycle Manager patch store. Check the network connectivity and firewall setup, and check the esxupdate logs for details.
  
  
• File Path: /var/run/log/esxupdate.log
Downloader:210 WARNING Download failed: ('http://<VC>:9084/vum/repository/hostupdate/vmw/vib20/loadesx/vibname.vib', '/tmp/esx-update-533431/vibname', 'Incomplete download from http://<VC>:9084/vum/repository/hostupdate/vmw/vib20/loadesx/vibname.vib, received 1276898 bytes, expecting 1574372 bytes'), 9 retry left.

Environment

• VMware vSphere ESXi 8.x, 9.x
• VMware vCenter Server 8.x, 9.x

Cause

When the client (the host) requests a file from the server (the vCenter), it reads the file in chunks from a buffer on the server. The file transfer happens in chunks based on the size of the file being downloaded. The server fills the file contents into a buffer sequentially and which is read by the client. Once the server fills in the last of the file contents into the buffer, it waits for about a second before closing the connection to the client. In good bandwidth networks, this time is quite sufficient for the client to read the contents of this buffer but in slow networks, the client is yet to completely read the buffer fully before the server closes the connection. This results in an incomplete download of the required host upgrade files and thus results in upgrade failure (via VUM or vLCM)

Resolution

The following resolution can be implemented after upgrading vCenter Server to vCenter Server 8.0 U3 and above

1. Connect to the vCenter Server through SSH, console, etc.

2. Take a backup of the json file before editing with the below command:

   cp /etc/vmware-rhttpproxy/listeners/edge_vum_9087https_9084http.json /var/tmp/edge_vum_9087https_9084http.json.bak

3. Edit /etc/vmware-rhttpproxy/listeners/edge_vum_9087https_9084http.json to follow the below instructions:

   Find the lines with the following text:
   "http_filters":  [

   Before these lines, add the following configuration:
   "delayed_close_timeout": 30,

   The final changes should look like this:
},
 "delayed_close_timeout": 30,
 "http_filters": [

4. Restart the rhttpproxy with the below command

   service-control --restart vmware-rhttpproxy

5. Wait a few minutes and attempt the host upgrade operation.

Follow the below guidance for versions below vCenter Server 8.0 U3

If you are already using vLCM for upgrades, set up a remote depot as indicated in Managing Host and Cluster Lifecycle and retry remediation.

The clusters might have different configurations and might have different vSphere Lifecycle Manager images. Hence, exporting the image from one cluster and uploading it to all remote depots might not work. To ensure that the remote depots have the correct content, ensure to export the image from the same cluster (on the vCenter) to which the depot override is being set and upload it to the remote depot specific to that cluster. For example, if there are five different clusters for which depot overrides are set, export the vSphere Lifecycle Manager image from each of these clusters and upload the offline bundles to the appropriate target remote depots. This ensures that all the software bits required for that specific cluster are present on the remote depot.

Once this is done, ensure the network between the hosts and the remote depot is good to have smooth file transfers.

If using VUM for upgrades, check if it is possible to switch to vLCM.

If there is an option to manually download the host upgrade files (offline bundles) elsewhere and transfer them to the host, the hosts can be upgraded via esxcli commands, refer to Upgrading Hosts by Using ESXCLI Commands