Downloading Esx metadata, vibs and vendor add-ons UMDS_SYNC task fails Message: Failed to execute UMDS command: /opt/vmware/vmware-umds/bin/vmware-umds -D --info-level error --task-id

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

In vSphere 9.X we can configure use the SDDC as a UMDS download repository, see documentation
https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/9-0/managing-host-and-cluster-lifecycle/working-with-vsphere-lifecycle-manager-depots/installing-setting-up-and-using-update-manager-download-service/setting-up-and-using-umds/setting-up-and-using-umd-1.html 

There is a scheduled automated task, that can also be triggered manually 

Fleet Management > Lifecycle -> Our VCF Instance > Binary Management > ESXi Components > Synchronize now.

 

This task runs for 1 or 2 days, and fails with a genetic error message

Failed to execute UMDS command: /opt/vmware/vmware-umds/bin/vmware-umds -D --info-level error --task-id ###########


Logs show error 403 [forbidden] and lists the URLs that fail 
403 indicates that the token used is not entitled for the download of said resource 

# grep -i vmware-downloadService /var/log/vmware/vcf/lcm/lcm-debug.log
LocalProcess INFO: <DATE && TIME> - <DATE && TIME> error vmware-downloadService[03605] [Originator@6876 sub=HostUpdateDepotManager] [vibDownloaderImpl 463] HTTP Error Code: 403
LocalProcess INFO: <DATE && TIME> - <DATE && TIME> error vmware-downloadService[03605] [Originator@6876 sub=HostUpdateDepotManager] [vibDownloaderImpl 473] Waiting for 15 seconds to retry.
LocalProcess INFO: <DATE && TIME> - <DATE && TIME> error vmware-downloadService[03416] [Originator@6876 sub=DownloadMgr] [downloadMgr 709] Executing download job {139737234037744} throws error: HTTP Error Code: 403
LocalProcess INFO: <DATE && TIME> - <DATE && TIME> error vmware-downloadService[03416] [Originator@6876 sub=DownloadMgr] [downloadMgr 817] Download failed for url: https://dl.broadcom.com/<TOKEN_HERE>/PROD/COMP/ESX_HOST/main/esx/vmw/vib20/vmkata/VMW_bootbank_vmkata_0.1-1vmw.700.1.0.15843807.vib

Environment

vSphere 9.0.X

Cause

The cause is the UMDS default config 
https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/9-0/managing-host-and-cluster-lifecycle/working-with-vsphere-lifecycle-manager-depots/installing-setting-up-and-using-update-manager-download-service/setting-up-and-using-umds/setting-up-and-using-umd-1.html 

By default the status lists ALL resources 
Run the following CLI as VCF user 

vcf@sddc-mgr [ ~ ]$ /opt/vmware/vmware-umds/bin/vmware-umds -G
Configured URLs
URL Type Removable URL
HOST     NO       https://dl.broadcom.com/<TOKEN_HERE>/PROD/COMP/ESX_HOST/main/vmw-depot-index.xml
HOST     NO       https://dl.broadcom.com/<TOKEN_HERE>/PROD/COMP/ESX_HOST/addon-main/vmw-depot-index.xml
HOST     NO       https://dl.broadcom.com/<TOKEN_HERE>/PROD/COMP/ESX_HOST/iovp-main/vmw-depot-index.xml
HOST     NO       https://dl.broadcom.com/<TOKEN_HERE>/PROD/COMP/ESX_HOST/vmtools-main/vmw-depot-index.xml
No Default URLs Configured

Patch store location  : /nfs/vmware/vcf/nfs-mount/umds/patch-store
Export store location :
Proxy Server          : Not configured

Host patch content download: enabled
Host Versions for which patch content will be downloaded:
esxio-8.0-INTL
esxio-9.0-INTL
embeddedEsx-7.0-INTL                      <= This source causes the error in this scenario
embeddedEsx-8.0-INTL
embeddedEsx-9.0-INTL

Resolution

Depending on configuration and entitlement - see https://knowledge.broadcom.com/external/article/395322/vcf-authenticated-downloads-token-troubl.html, the Token/ Site ID might not be entitled for 7.X or 8.X.

To stop the download for a specific Product version - example below will be 7.X - please follow the process described below:
 

1. View configuration of UMDS

vcf@sddc-mgr [ ~ ]$ /opt/vmware/vmware-umds/bin/vmware-umds -G
Configured URLs
URL Type Removable URL
HOST     NO       https://dl.broadcom.com/<TOKEN_HERE>/PROD/COMP/ESX_HOST/main/vmw-depot-index.xml
HOST     NO       https://dl.broadcom.com/<TOKEN_HERE>/PROD/COMP/ESX_HOST/addon-main/vmw-depot-index.xml
HOST     NO       https://dl.broadcom.com/<TOKEN_HERE>/PROD/COMP/ESX_HOST/iovp-main/vmw-depot-index.xml
HOST     NO       https://dl.broadcom.com/<TOKEN_HERE>/PROD/COMP/ESX_HOST/vmtools-main/vmw-depot-index.xml
No Default URLs Configured

Patch store location  : /nfs/vmware/vcf/nfs-mount/umds/patch-store
Export store location :
Proxy Server          : Not configured

Host patch content download: enabled
Host Versions for which patch content will be downloaded:
esxio-8.0-INTL
esxio-9.0-INTL
embeddedEsx-7.0-INTL
embeddedEsx-8.0-INTL
embeddedEsx-9.0-INTL

2. List what Host platforms are being downloaded 
vcf@sddc-mgr [ ~ ]$  /opt/vmware/vmware-umds/bin/vmware-umds --list-host-platforms
Supported ESX Host platforms:
esxio-8.0-INTL
esxio-9.0-INTL
embeddedEsx-7.0-INTL
embeddedEsx-8.0-INTL
embeddedEsx-9.0-INTL

Disable 7.X 
vcf@sddc-mgr [ ~ ]$ /opt/vmware/vmware-umds/bin/vmware-umds -S -d embeddedEsx-7.0.0

Setting up UMDS configuration
Host update downloads for platform embeddedEsx-7.0.0: Disabled
**********************************************
Command SUCCEEDED
**********************************************

vcf@sddc-mgr [ ~ ]$  /opt/vmware/vmware-umds/bin/vmware-umds --list-host-platforms
Supported ESX Host platforms:
esxio-8.0-INTL
esxio-9.0-INTL
embeddedEsx-8.0-INTL
embeddedEsx-9.0-INTL

3. restart LCM to stop download task 
# systemctl restart lcm

4. Manually trigger this task by following teh path 
Fleet Management > Lifecycle -> Our VCF Instance > Binary Management > ESXi Components > Synchronize now.