"Cannot download VIB" error upgrading to ESXi 7.0 Update 2 using vLCM

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• You are trying to remediate ESXi host to 7.0 Update 2 using vLCM / Update Manager fails, you see error message similar to:

Cannot download VIB: ''. This might be because of network issues or the specified VIB does NOT exist or does NOT have a proper 'read' privilege set. Please make sure the specified VIB exists and is accessible from vCenter Server.

• In the /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server-log4cpp.log file, you will see error similar to:

[2021-03-12 14:29:32:727 'DownloadMgr' 139972188141312 ERROR]  [downloadMgr, 678] Executing download job {139970836824816} throws error: HTTP Error Code: 404
[2021-03-12 14:29:32:727 'DownloadMgr' 139972188141312 ERROR]  [downloadMgr, 767] Download failed for url: https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/esx-update/VMware_bootbank_esx-update_7.0.2-0.0.17630552.vib
[2021-03-12 14:29:32:727 'HostUpdateDepotManager' 139971269633792 ERROR]  [vibDownloaderImpl, 453] Downloading file failed for: https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/esx-update/VMware_bootbank_esx-update_7.0.2-0.0.17630552.vib
...
[2021-03-12 14:29:32:727 'HostUpdateDepotManager' 139971269633792 ERROR]  [patchDepotManager, 621] Failed to download 1 vib: https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/esx-update/VMware_bootbank_esx-update_7.0.2-0.0.17630552.vib
[2021-03-12 14:29:32:728 'HostUpdateDepotManager' 139971269633792 ERROR]  [installController, 361] Install on host: <host fqdn>(Entity: host-<moid>) failed. Error: Failed to download 1 vib: https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/esx-update/VMware_bootbank_esx-update_7.0.2-0.0.17630552.vib
[2021-03-12 14:29:32:795 'SingleHostRemediateTask.SingleHostRemediateTask{50}' 139971269633792 ERROR]  [singleHostRemediateTask, 286] Remediation error: Failed to download 1 vib: https://hostupdate.vmware.com/software/VUM/PRODUCTION/main
/esx/vmw/vib20/esx-update/VMware_bootbank_esx-update_7.0.2-0.0.17630552.vib

Note: The preceding log excerpts are only examples. The date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSphere ESXi 7.0.x

Cause

This issue occurs on vCenter Server deployments prior to March 12, 2021, due to ESXi 7.0 U2 depot.zip from offline and online sites removal to address a critical issue tracked in KB Failed to load crypto64.efi Fatal error: 15 (Not found) after patching ESXi host to 7.0 Update 2. 

vLCM / Update Manager will still have the Metadata of updates as it automatically downloads it, as a result vLCM is showing 7.0 U2 as an available upgrade patch.

Resolution

This issue is resolved in ESXi 7.0 Update 2a. For more information, see VMware ESXi 7.0 Update 2a Release Notes.

VMware recommends to NOT upgrade to 7.0 U2 and upgrading to 7.0 U2a instead.

Additional Information

• vCenter Servers deployed on or after March 12, 2021, there are no impacts. As the patch has been removed before your vLCM depot synched with Patch Stores.
• vCenter Servers deployed before March 12, 2021, ESXi 7.0 U2 metadata is already downloaded into your depot and hence the listing for removed patch content (bulletins with build number 17630552) would exists. Scan would pass against the baselines of 7.0 U2. Remediation fails with the following error mentioned in Symptoms section of this KB 
• If you have already staged the content of ESXi 7.0 U2, the remediation would succeed for the patch baseline on BIOS machines. If update is tried on UEFI would hit into issue outlined in Failed to load crypto64.efi Fatal error: 15 (Not found) after patching ESXi host to 7.0 Update 2
• If you have configured UMDS for offline patching, patch content (bulletins with build number 17630552) continues to persist in the UMDS depot directory even after roll-back and can be patched successfully. It is recommended that you manually remove the binaries/VIBs from the UMDS directory or direct the downloads to the new UMDS directory location to be in-sync with VMware’s online depot. For more information, see How to configure the UMDs download location
• Stateless boot through Autodeploy will have to update the existing rule to point to the previously release image profile and wait for the updated patch for ESXi 7.0 U2 specific content
• If you are managing ESXi image at a cluster level, ESXi 7.0 U2 baseimage were removed and you will have to wait for an updated patch to be released.
• You may still try the fresh deployment of 7.0 U2 using the installer ISO, see the following for information on how to download Download Broadcom Products and Software