Error: A general system error occurred: Failed to download VIB(s): Error: HTTP Error Code: 403, vLCM fails to download the ESXi patches and images from online repositories

Products

VMware vSphere ESXi VMware vCenter Server VMware SDDC Manager

Issue/Introduction

vCenter Lifecycle Management (vLCM) fails to download ESXi patches/images from VMware by Broadcom public repositories with following error message:

A general system error occurred:
Failed to download VIB(s): URL:
https://hostupdate.broadcom.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/<vib_name>.vib
Error: HTTP Error Code: 403

Connectivity Status under Lifecycle Manager > Settings > Patch Setup for hostupdate.vmware.com URL's shows Enabled status as Yes and Connection Status as Not Connected or Validating.

Patching ESXi through vCenter Server UI fails when download repository is configured as https://hostupdate.vmware.com/software/VUM/PRODUCTION/<>/vmw-depot-index.xml with error:

"Cannot download VIB: ''. This might be because of network issues or the specified VIB does NOT exist or does NOT have a proper 'read' privilege set. Please make sure the specified VIB exists and is accessible from vCenter Server."

The /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log displays the error:

YYYY-MM-DDTHH:MM:SS info vmware-vum-server[1069416] [Originator@6876 sub=httpDownload] [httpDownloadPosix 769] curl_easy_perform() succeeded - url: https://hostupdate.vmware.com/software/VUM/PRODUCTION/vmtools-main/vmw-depot-index.xml
YYYY-MM-DDTHH:MM:SS error vmware-vum-server[1069416] [Originator@6876 sub=DownloadMgr] [downloadMgr 705] Executing download job {140344436625712} throws error: HTTP Error Code: 403
YYYY-MM-DDTHH:MM:SS info vmware-vum-server[1069416] [Originator@6876 sub=DownloadMgr] [downloadMgr 807] Download failed but destination file /tmp/vciHB8meY exists and is valid. Ignoring error
...
YYYY-MM-DDTHH:MM:SS error vmware-vum-server[1069421] [Originator@6876 sub=Default] [updateDownloaderImpl 116] File download error: Downloading file: https://hostupdate.vmware.com/software/VUM/PRODUCTION/vmtools-main/vmw-depot-index.xml failed, 0 byte downloaded.

vLCM may also generate the following error:
A general system error occurred: A depot is inaccessible or has invalid contents. Make sure an official depot source is used and verify connection to the depot.

Environment

VMware vCenter 7.x
VMware vCenter 8.x
VMware ESXi 7.x
VMware ESXi 8.x

Cause

Public facing repository URLs and authentication mechanisms are changing. Download URLs are no longer common but unique for each customer therefore will require to be re-configured.

Resolution

Download Token from Broadcom Support Portal by following the steps in VCF Authenticated downloads Configuration Update Instructions. If you already have it, apply the same by following below instructions.
Disable the default repositories and configure the repositories with the Token

Note: Scripted method to update the URLs are available in Authenticated Download Configuration Update Script.
1. Log into vSphere client
2. On the vSphere Client Home page, Navigate to Lifecycle Manager.
3. Click 'settings' tab. Under 'settings' tab,
4. Go to Administration → Patch Setup.
5. Disable default vLCM URLs by clicking "Disable" button:
  
  Note: If the vCenter Server is running on 7.0 U3c (build 19234570) or 8.0 GA versions (8.0GA, 8.0a, 8.0b & 8.0c), configuring the Custom URLs mentioned below will fail. Please perform Update Manager DB reset before proceeding with below steps if the vCenter Server is on these impacted versions.
  
  Note: The default URLs cannot be removed and delete button will be greyed out, it can only be Disabled.
6. Add the below URLs one by one by clicking 'NEW':
  
  New URLs:
  
  https://dl.broadcom.com/<Download Token>/PROD/COMP/ESX_HOST/main/vmw-depot-index.xml https://dl.broadcom.com/<Download Token>/PROD/COMP/ESX_HOST/addon-main/vmw-depot-index.xml https://dl.broadcom.com/<Download Token>/PROD/COMP/ESX_HOST/iovp-main/vmw-depot-index.xml https://dl.broadcom.com/<Download Token>/PROD/COMP/ESX_HOST/vmtools-main/vmw-depot-index.xml
  
  Note: Replace <Download Token> with the actual downloaded token id.
7. Ensure to remove any of the old urls that are being replaced as leaving these will cause vCenter to continue to try and reach them resulting in errors.
8. Restart update manager service from an SSH session to vCenter Server
  
  service-control --restart vmware-updatemgr
9. Download the content by going to vSphere Client Home page → Lifecycle Manager and selecting "ACTIONS" → Updates → Sync updates.
  
  Notes:
  1. Please wait for the Sync updates operation to complete before initiating the ESXi updates.
  2. If vLCM/VUM is trying to download the VIBs using an "older/inactive token" or still using "hostupdate.vmware.com" even after updating the right token, please reset the vLCM Database by following the KB Resetting the VMware Update Manager Database.
    
    Following are couple of sample scenarios where vLCM DB Reset is required.
    
    Example - 1: ESXi Patching is failing with below error message and the "<AuthenticationToken>" in the URL is an older token which is no more valid.
    
    Cannot download VIB: https://dl.broadcom.com/<AuthenticationToken>/PROD/COMP/ESX_HOST/main/esx/vmw/vib20/vibDir/vibnameHere.vib. This might be because of network issues or the specified VIB does NOT exist or does NOT have a proper 'read' privilege set. Please make sure the specified VIB exists and is accessible from vCenter Server.
    
    Example - 2: ESXi Patching is failing with below error message and vLCM is still using the old default URL hostupdate.vmware.com even though those default URLs are disabled.
    
    Cannot download VIB: Cannot download VIB 'https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/vibDir/vibnameHere.vib'. This might be because of network issues or the specified VIB does NOT exist or does NOT have a proper 'read' privilege set. Make sure the specified VIB exists and is accessible from vCenter Server.
10. Follow the steps from Error: "The vCenter Server is not able to reach the specified URL" vCenter Server patching via VAMI fails to download the updates from online repositories
  to update VAMI URL