SDDC Manager fail to download bundles with error: DepotConnectionFailureException: Error downloading bundle [/COMP/SDDC_MANAGER_VCF/bundles/bundle-202281.tar] from dl.broadcom:443

Products

• VMware SDDC Manager

Issue/Introduction

In SDDC Manager - Bundle Management you see the following error:

DepotConnectionFailureException: Error downloading bundle [/COMP/SDDC_MANAGER_VCF/bundles/bundle-202281.tar] from dl.broadcom:443

In the LCM log you see the following:

/var/log/vmware/vcf/lcm/lcm-debug.log:

DEBUG [vcf_lcm,67e3a9ad96b887b4d86fe9038062db40,6a4e] [c.v.e.s.l.b.d.d.DepotBundleDownloadServiceImpl,Scheduled-8] Downloading index file from remote server fileName:/COMP/SDDC_MANAGER_VCF/index.v3
DEBUG [vcf_lcm,67e3a9ad96b887b4d86fe9038062db40,6a4e] [c.v.e.s.l.b.d.depot.DepotDownloader,Scheduled-8] Getting file size for [/COMP/SDDC_MANAGER_VCF/index.v3] from URL[https://dl.broadcom.com:443/<DOWNLOAD_TOKEN>/PROD/COMP/SDDC_MANAGER_VCF/index.v3]
DEBUG [vcf_lcm,67e3a9ad96b887b4d86fe9038062db40,6a4e] [c.v.e.s.l.b.d.depot.DepotDownloader,Scheduled-8] Executing HEAD //PROD/COMP/SDDC_MANAGER_VCF/index.v3
DEBUG [vcf_lcm,67e3a9ad96b887b4d86fe9038062db40,6a4e] [c.v.e.s.l.b.d.depot.DepotDownloader,Scheduled-8] Getting file for [/COMP/SDDC_MANAGER_VCF/index.v3] from URL[https://dl.broadcom.com:443/<DOWNLOAD_TOKEN>/PROD/COMP/SDDC_MANAGER_VCF/index.v3]
DEBUG [vcf_lcm,67e3a9ad96b887b4d86fe9038062db40,6a4e] [c.v.e.s.l.b.d.depot.DepotDownloader,Scheduled-8] Executing GET /<DOWNLOAD_TOKEN>/PROD/COMP/SDDC_MANAGER_VCF/index.v3
java.io.FileNotFoundException: /nfs/vmware/vcf/nfs-mount/bundle/depot/local/index.v3 (Permission denied)

 

Environment

VCF 5.x

Cause

Permissions on the underlying folder structure is incorrect in SDDC Manager.

Expected permissions:
/nfs/vmware/vcf/nfs-mount/bundle/depot/local/    drwxr-xr-x 5 vcf_lcm vcf 
/nfs/vmware/vcf/nfs-mount/bundle/depot/          drwxr-xr-x 3 vcf_lcm vcf 
/nfs/vmware/vcf/nfs-mount/bundle/                drwxrwxr-x 23 vcf_lcm vcf
/nfs/vmware/vcf/nfs-mount/                       drwxrwxr-x 10 root vcf
/nfs/vmware/vcf/                                 drwxr-xr-x 3 root root

Incorrect permissions (Example):
# ls -l /nfs/vmware/vcf/nfs-mount/bundle/depot/local/
drwxr-xr-x 2 vcf vcf  4096 Sep 20  2024 bundles
drwxr-xr-x 2 vcf vcf 20480 Feb 21 13:47 manifests
... data omitted

# ls -l /nfs/vmware/vcf/nfs-mount/bundle/depot/
drw-r----- 4 vcf_lcm vcf 4096 Feb 21 13:47 local

# ls -l /nfs/vmware/vcf/nfs-mount/bundle/
drwxr-xr-x 3 vcf_lcm vcf 4096 May 30  2024 0602cf8f-9a37-####-####-############
drwxr-xr-x 3 vcf_lcm vcf 4096 May 30  2024 084d1af4-7bfb-####-####-############

# ls -l /nfs/vmware/vcf/nfs-mount/
drwxrwx---  8 vcf     vcf      4096 Apr  2 13:33 apToolBundles
drwxrwxr-x  5 vcf_lcm backup   4096 Mar 15  2024 backup
... data omitted

# ls -l /nfs/vmware/vcf/
drwxrwxr-x 12 root vcf 4096 Apr  1 14:04 nfs-mount

Resolution

1. Take a snapshot of the SDDC Manager
   
   
2. To verify if the permissions are correct on the folders, run the following commands and make a note of the output:
   ls -l /nfs/vmware/vcf/nfs-mount/bundle/depot/local/
ls -l /nfs/vmware/vcf/nfs-mount/bundle/depot/
ls -l /nfs/vmware/vcf/nfs-mount/bundle/
ls -l /nfs/vmware/vcf/nfs-mount/
ls -l /nfs/vmware/vcf/
   
   
3. Compare this output to the Expected permission as mentioned. If the output is different from the expected permission, change the permissions as follows:
   chmod 775 -R /nfs/vmware/vcf/nfs-mount/bundle/
chown vcf_lcm:vcf /nfs/vmware/vcf/nfs-mount/bundle/