Cluster remediation fails with dl.broadcom token URL

Products

VMware vSphere ESXi

Issue/Introduction

Token URLs have been generated and configured on vSphere Component - See KB https://knowledge.broadcom.com/external/article/390098
Following the creation of the new token in the Broadcom Support, the previous token is marked as inactive but the URL still contains the old inactive token causing the Cluster remediation to fail

Example:

Cannot download VIB 'https://dl.broadcom.com/<Inactive Token>/PROD/COMP/ESX_HOST/main/esx/vmw/vib20/lpfc/VMW_bootbank_lpfc_14.4.0.40-35vmw.803.0.70.24674464.vib' This might be because of network issues or the specified VIB does NOT exist or does NOT have a proper 'read' privilege set. Make sure the specified VIB exists and is accessible from vCenter Server.

Environment

ESXi 7.x
ESXi 8.x
vCenter 7.x
vCenter 8.x

Cause

Previously generated tokens are still being retained in the VUM DB when trying to download VIBs

/var/log/vmware/vmware-updatemgr/vum-server/vum-server.log

The depot is connected with the correct token URLs

2025-05-13T13:32:57.624+02:00 info vmware-vum-server[#####] [Originator@#### sub=ConfigurationMgr] [configurationMgr 1750] Depot URL type check: https://dl.broadcom.com/<active token>/PROD/COMP/ESX_HOST/main/vmw-depot-index.xml is a valid URL containing 4.x+ host update content
2025-05-13T13:32:57.708+02:00 info vmware-vum-server[#####] [Originator@#### sub=ConfigurationMgr] [configurationMgr 1750] Depot URL type check: https://dl.broadcom.com/<active token>/PROD/COMP/ESX_HOST/addon-main/vmw-depot-index.xml is a valid URL containing 4.x+ host update content
2025-05-13T13:32:57.816+02:00 info vmware-vum-server[#####] [Originator@#### sub=ConfigurationMgr] [configurationMgr 1750] Depot URL type check: https://dl.broadcom.com/<active token>/PROD/COMP/ESX_HOST/iovp-main/vmw-depot-index.xml is a valid URL containing 4.x+ host update content
2025-05-13T13:36:10.789+02:00 info vmware-vum-server[#####] [Originator@#### sub=ConfigurationMgr] [configurationMgr 1750] Depot URL type check: https://dl.broadcom.com/<active token>/PROD/COMP/ESX_HOST/vmtools-main/vmw-depot-index.xml is a valid URL containing 4.x+ host update content

The vibs are still retaining the previously used inactive token

2025-05-13T11:26:45.429+02:00 verbose vmware-vum-server[####] [Originator@#### sub=httpDownload opID=#####] [httpDownloadPosix ####] The url https://dl.broadcom.com/<inactive token>/PROD/COMP/ESX_HOST/main/esx/vmw/vib20/lpfc/VMW_bootbank_lpfc_14.4.0.40-35vmw.803.0.70.24674464.vib is inaccessible.

Resolution

1. Take snapshot of vCenter
2. Reset the VUMDB following the process below

Note** If your cluster is an NSX Cluster use the workaround section below to reset the VUMDB

3. Confirm the URLs with the correct token are still present in your Patch Setup configuration
4. You may need to re-add & validate the image
5. Retry remediating the cluster

Process to Reset the VMware Update Manager Database on a VCSA 7.0, or 8.0

Caution: Resetting the Update Manager database is a destructive task. Custom baselines (but not Cluster Images), custom download settings and manually imported patches/ISOs will be removed and will need to be reapplied following the reset. Before applying the steps below, take a backup or an offline-snapshot (in powered-off state) of the vCenter Server. If the vCenter Server is part of an ELM environment, take a snapshot or a backup of all vCenters within the ELM domain. Note all of the custom configurations within Update Manager - e.g. proxy settings, third party download URLs, customized baselines, etc. - before proceeding.

SSH to the vCenter via root
If not in BASH shell, switch to the BASH Shell
```
shell
```
Stop the VMware Update Manager Service
```
service-control --stop vmware-updatemgr
```
Run the following commands to reset the VMware Update Manager Database:
```
python /usr/lib/vmware-updatemgr/bin/updatemgr-utility.py reset-db
```
Delete the contents of the VMware Update Manager Patch Store
```
rm -rf /storage/updatemgr/patch-store/*
```

Start the VMware Update Manager Service

service-control --start vmware-updatemgr

Notes

Logging out and back into the vSphere Web Client may be required to see the change.
For vSAN environments, this will also remove the vSAN default baselines. These baselines are re-created automatically when there is a configuration change to vSAN such as add/remove a host/disk or an update to the HCL DB. The vSAN cluster can still be updated without the vSAN default baselines.
Since the new "download token"-based URLs are considered custom URLs, they will need to be re-added after the reset. Refer to KB VCF authenticated downloads configuration update instructions to update the necessary systems.

Workaround

To reset the Update Manager database if NSX-T is installed in a vLCM-enabled cluster (applicable for vCenter Server Appliance 7.0 U1 and beyond).

After running the reset-db script, perform the following steps.

Create an empty vLCM-enabled cluster
1. Log into vCenter
2. Right click on vCenter in the Inventory tab.
3. Click New Cluster
4. Make sure to select "Import image from existing host" to avoid having to add a host to the cluster.
From the NSX Manager UI, configure NSX on the cluster created in step 1. This will ensure the NSX depot gets uploaded to the VC again.
1. In the NSX UI, go to the Hosts and Clusters tab.
2. Locate the new empty cluster and check the box next to it.
3. Find the New empty cluster, and put a check next to it.
4. Click Add NSX at the top. Select the Transport Node Profile to use. This does NOT affect live traffic as the cluster is an empty cluster.
The cluster created in step 1 can now be deleted. After the related tasks in vCenter show complete.

The NSX depot gets uploaded to vCenter when NSX is configured on a vLCM cluster only if the depot is not already present. So, if the reset-db script removes this depot from vCenter, then this depot can be uploaded again by creating an empty vLCM cluster and enabling NSX on that cluster.