Host upgrade fails in low bandwidth Remote Office and Branch Office (ROBO) setup

Products

• VMware vSphere ESXi

Issue/Introduction

This article helps you to understand the failures that might arise in ROBO environments when the network bandwidth is very low and try out potential solutions.

Symptoms:
If the hosts are in a ROBO (Remote Office/Branch Office) setup and have low bandwidth (< 5 Mbps), upgrading these hosts using VUM or vLCM results in a failure. The errors that can be seen are listed below:

During the vLCM upgrade:
An error occurred while downloading VIB from the URL http://<VC>:9084/vum/repository/hostupdate/vmw/vib20/loadesx/<vibname>.vib

During the VUM upgrade:
The host cannot download files from the VMware vSphere Lifecycle Manager patch store. Check the network connectivity and firewall setup, and check the esxupdate logs for details.

There should be logs like these on the host:

Downloader:210 WARNING Download failed: ('http://<VC>:9084/vum/repository/hostupdate/vmw/vib20/loadesx/vibname.vib', '/tmp/esx-update-533431/vibname', 'Incomplete download from http://<VC>:9084/vum/repository/hostupdate/vmw/vib20/loadesx/vibname.vib, received 1276898 bytes, expecting 1574372 bytes'), 9 retry left.

Environment

VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0

Cause

When the client (the host) requests for a file from the server (the vCenter), it reads the file in chunks from a buffer on the server. The file transfer happens in chunks based on the size of the file being downloaded. The server fills the file contents into a buffer sequentially and which is read by the client. Once the server fills in the last of the file contents into the buffer, it waits for about a second before closing the connection to the client. In good bandwidth networks, this time is quite sufficient for the client to read the contents of this buffer but in slow networks, the client is yet to completely read the buffer fully before the server closes the connection. This results in an incomplete download of the required host upgrade files and thus results in upgrade failure (via VUM or vLCM)

Resolution

The following resolution can be implemented after upgrading vCenter Server to vCenter Server 8.0 U3

1. Connect to the vCenter Server through SSH, console, etc.. 
   
   
2. Take a backup of the json file before editing with the below command:
   
   cp /etc/vmware-rhttpproxy/endpoints.conf.d/edge_vum_9087https_9084http.json /var/tmp/edge_vum_9087https_9084http.json.bak

3. Edit /etc/vmware-rhttpproxy/endpoints.conf.d/edge_vum_9087https_9084http.json to follow the below instructions:
   
   Find the lines with the following text:
   "http_filters":  [

   Before these lines, please add the following configuration:
   "delayed_close_timeout": 30,

   The final changes should look like this:
   },
 "delayed_close_timeout": 30,
 "http_filters": [
  {

4. Restart the rhttpproxy with the below command
   
   service-control --restart vmware-rhttpproxy

5. Wait a few minutes and attempt the host upgrade operation.

Follow the below guidance for versions below vCenter Server 8.0 U3

If already using vLCM for upgrades, please set up a remote depot as indicated in Manage Depot Overrides for a Cluster and retry remediation.

The clusters might have different configurations and might have different vSphere Lifecycle Manager images. Hence, exporting the image from one cluster and uploading it to all remote depots might not work. To ensure that the remote depots have the correct content, please ensure to export the image from the same cluster (on the vCenter) to which the depot override is being set and upload it to the remote depot specific to that cluster. For example, if there are five different clusters for which depot overrides are set, please export the vSphere Lifecycle Manager image from each of these clusters and upload the offline bundles to the appropriate target remote depots. This ensures that all the software bits required for that specific cluster are present on the remote depot.

Once this is done ensure the network between the hosts and the remote depot is good to have smooth file transfers.

If using VUM for upgrades, please check if it is possible to switch to vLCM.

If there an option to manually download the host upgrade files (offline bundles) elsewhere and transfer them to the host, the hosts can be upgraded via esxcli commands, please refer to ESXi 7.0 ESXCLI Command Reference