SDDC Manager fail to download bundles with error: DepotConnectionFailureException: Error downloading bundle [/COMP/SDDC_MANAGER_VCF/bundles/bundle-202281.tar] from dl.broadcom:443

Products

• VMware SDDC Manager

Issue/Introduction

In SDDC Manager - Bundle Management you see the following error:

DepotConnectionFailureException: Error downloading bundle [/COMP/SDDC_MANAGER_VCF/bundles/bundle-202281.tar] from dl.broadcom:443

In the LCM log you see the following:

/var/log/vmware/vcf/lcm/lcm-debug.log:

DEBUG [vcf_lcm,####################################,####] [c.v.e.s.l.b.d.d.DepotBundleDownloadServiceImpl,Scheduled-8] Downloading index file from remote server fileName:/COMP/SDDC_MANAGER_VCF/index.v3
DEBUG [vcf_lcm,####################################,####] [c.v.e.s.l.b.d.depot.DepotDownloader,Scheduled-8] Getting file size for [/COMP/SDDC_MANAGER_VCF/index.v3] from URL[https://dl.broadcom.com:443/<DOWNLOAD_TOKEN>/PROD/COMP/SDDC_MANAGER_VCF/index.v3]
DEBUG [vcf_lcm,####################################,####] [c.v.e.s.l.b.d.depot.DepotDownloader,Scheduled-8] Executing HEAD //PROD/COMP/SDDC_MANAGER_VCF/index.v3
DEBUG [vcf_lcm,####################################,####] [c.v.e.s.l.b.d.depot.DepotDownloader,Scheduled-8] Getting file for [/COMP/SDDC_MANAGER_VCF/index.v3] from URL[https://dl.broadcom.com:443/<DOWNLOAD_TOKEN>/PROD/COMP/SDDC_MANAGER_VCF/index.v3]
DEBUG [vcf_lcm,####################################,####] [c.v.e.s.l.b.d.depot.DepotDownloader,Scheduled-8] Executing GET /<DOWNLOAD_TOKEN>/PROD/COMP/SDDC_MANAGER_VCF/index.v3
java.io.FileNotFoundException: /nfs/vmware/vcf/nfs-mount/bundle/depot/local/index.v3 (Permission denied)

 

Environment

VCF 5.x

VCF 9.x

Cause

Permissions on the underlying folder structure is incorrect in SDDC Manager.

Expected permissions:
#ls -l /nfs/vmware/vcf/nfs-mount/bundle/depot/local/   
drwxr-xr-x 5 vcf_lcm vcf 

#ls -l /nfs/vmware/vcf/nfs-mount/bundle/depot/         
drwxr-xr-x 3 vcf_lcm vcf 

#ls -l /nfs/vmware/vcf/nfs-mount/bundle/               
drwxrwxr-x 23 vcf_lcm vcf

#ls -l /nfs/vmware/vcf/nfs-mount/                       
drwxrwxr-x 10 root vcf

#ls -l /nfs/vmware/vcf/                                 
drwxr-xr-x 3 root root

Incorrect permissions (Example):
# ls -l /nfs/vmware/vcf/nfs-mount/bundle/depot/local/
drwxr-xr-x 2 vcf vcf  4096 Sep 20  2024 bundles
drwxr-xr-x 2 vcf vcf 20480 Feb 21 13:47 manifests
... data omitted

# ls -l /nfs/vmware/vcf/nfs-mount/bundle/depot/
drw-r----- 4 vcf_lcm vcf 4096 Feb 21 13:47 local

# ls -l /nfs/vmware/vcf/nfs-mount/bundle/
drwxr-xr-x 3 vcf_lcm vcf 4096 May 30  2024 0602cf8f-9a37-####-####-############
drwxr-xr-x 3 vcf_lcm vcf 4096 May 30  2024 084d1af4-7bfb-####-####-############

# ls -l /nfs/vmware/vcf/nfs-mount/
drwxrwx---  8 vcf     vcf      4096 Apr  2 13:33 apToolBundles
drwxrwxr-x  5 vcf_lcm backup   4096 Mar 15  2024 backup
... data omitted

# ls -l /nfs/vmware/vcf/
drwxrwxr-x 12 root vcf 4096 Apr  1 14:04 nfs-mount

Resolution

1. Take a snapshot of the SDDC Manager
   
   
2. To verify if the permissions are correct on the folders, run the following commands and make a note of the output:
   ls -l /nfs/vmware/vcf/nfs-mount/bundle/depot/local/
ls -l /nfs/vmware/vcf/nfs-mount/bundle/depot/
ls -l /nfs/vmware/vcf/nfs-mount/bundle/
ls -l /nfs/vmware/vcf/nfs-mount/
ls -l /nfs/vmware/vcf/
   
   
3. Compare this output to the Expected permission as mentioned. If the output is different from the expected permission, change the permissions as follows:
   chmod 775 -R /nfs/vmware/vcf/nfs-mount/bundle/
chown -R vcf_lcm:vcf /nfs/vmware/vcf/nfs-mount/bundle/


4. To retry the download, Under Bundle Management, Bundles, Bundle Details, click Retry Download.

Additional Information

If you still see the Error follow KB to Cleanup up bundles  : DepotConnectionFailureException: Error downloading bundle [/COMP/SDDC_MANAGER_VCF/bundles/bundle-<TBD>.tar] from dl.broadcom:443

In some upgraded environments(IE 9.0.1 and 5.2.2) the nfs-mount folder will show nginx as the user. This is a known issue and will be fixed in a future version.