SDDC Manager fail to download bundles with error: DepotConnectionFailureException: Error downloading bundle [/COMP/SDDC_MANAGER_VCF/bundles/bundle-202281.tar] from dl.broadcom:443

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

In SDDC Manager - Bundle Management you see the following error:

DepotConnectionFailureException: Error downloading bundle [/COMP/SDDC_MANAGER_VCF/bundles/bundle-202281.tar] from dl.broadcom:443

In the LCM log you see the following:

/var/log/vmware/vcf/lcm/lcm-debug.log:

DEBUG [vcf_lcm,####################################,####] [c.v.e.s.l.b.d.d.DepotBundleDownloadServiceImpl,Scheduled-8] Downloading index file from remote server fileName:/COMP/SDDC_MANAGER_VCF/index.v3
DEBUG [vcf_lcm,####################################,####] [c.v.e.s.l.b.d.depot.DepotDownloader,Scheduled-8] Getting file size for [/COMP/SDDC_MANAGER_VCF/index.v3] from URL[https://dl.broadcom.com:443/<DOWNLOAD_TOKEN>/PROD/COMP/SDDC_MANAGER_VCF/index.v3]
DEBUG [vcf_lcm,####################################,####] [c.v.e.s.l.b.d.depot.DepotDownloader,Scheduled-8] Executing HEAD //PROD/COMP/SDDC_MANAGER_VCF/index.v3
DEBUG [vcf_lcm,####################################,####] [c.v.e.s.l.b.d.depot.DepotDownloader,Scheduled-8] Getting file for [/COMP/SDDC_MANAGER_VCF/index.v3] from URL[https://dl.broadcom.com:443/<DOWNLOAD_TOKEN>/PROD/COMP/SDDC_MANAGER_VCF/index.v3]
DEBUG [vcf_lcm,####################################,####] [c.v.e.s.l.b.d.depot.DepotDownloader,Scheduled-8] Executing GET /<DOWNLOAD_TOKEN>/PROD/COMP/SDDC_MANAGER_VCF/index.v3
java.io.FileNotFoundException: /nfs/vmware/vcf/nfs-mount/bundle/depot/local/index.v3 (Permission denied)

Additional symptoms:

SDDC Manager displays an error in the UI when attempting to download bundles.
In the logs, you will see either DepotConnectionFailureException or BundleRepoWriteFailureException indicating SDDC Manager cannot write to the NFS mount.

Environment

VCF 5.x

VCF 9.x

Cause

Permissions on the underlying folder structure is incorrect in SDDC Manager.

Expected permissions:
#ls -l /nfs/vmware/vcf/nfs-mount/bundle/depot/local/   
drwxr-xr-x 5 vcf_lcm vcf 

#ls -l /nfs/vmware/vcf/nfs-mount/bundle/depot/         
drwxr-xr-x 3 vcf_lcm vcf 

#ls -l /nfs/vmware/vcf/nfs-mount/bundle/               
drwxrwxr-x 23 vcf_lcm vcf

#ls -l /nfs/vmware/vcf/nfs-mount/                       
drwxrwxr-x 10 root vcf

#ls -l /nfs/vmware/vcf/                                 
drwxr-xr-x 3 root root

Incorrect permissions (Example):
# ls -l /nfs/vmware/vcf/nfs-mount/bundle/depot/local/
drwxr-xr-x 2 vcf vcf  4096 Sep 20  2024 bundles
drwxr-xr-x 2 vcf vcf 20480 Feb 21 13:47 manifests
... data omitted

# ls -l /nfs/vmware/vcf/nfs-mount/bundle/depot/
drw-r----- 4 vcf_lcm vcf 4096 Feb 21 13:47 local

# ls -l /nfs/vmware/vcf/nfs-mount/bundle/
drwxr-xr-x 3 vcf_lcm vcf 4096 May 30  2024 0602cf8f-9a37-####-####-############
drwxr-xr-x 3 vcf_lcm vcf 4096 May 30  2024 084d1af4-7bfb-####-####-############

# ls -l /nfs/vmware/vcf/nfs-mount/
drwxrwx---  8 vcf     vcf      4096 Apr  2 13:33 apToolBundles
drwxrwxr-x  5 vcf_lcm backup   4096 Mar 15  2024 backup
... data omitted

# ls -l /nfs/vmware/vcf/
drwxrwxr-x 12 root vcf 4096 Apr  1 14:04 nfs-mount

Resolution

1. Take a snapshot of the SDDC Manager
   
   
2. To verify if the permissions are correct on the folders, run the following commands and make a note of the output:
   ls -l /nfs/vmware/vcf/nfs-mount/bundle/depot/local/
ls -l /nfs/vmware/vcf/nfs-mount/bundle/depot/
ls -l /nfs/vmware/vcf/nfs-mount/bundle/
ls -l /nfs/vmware/vcf/nfs-mount/
ls -l /nfs/vmware/vcf/
   
   
3. Compare this output to the Expected permission as mentioned. If the output is different from the expected permission, change the permissions as follows:
   
   1. Correct the parent directory ownership:
      chown root:root /nfs/vmware/vcf/
   2. Correct the bundle subdirectory ownership and permissions:
      chown -R vcf_lcm:vcf /nfs/vmware/vcf/nfs-mount/bundle/
chmod 775 -R /nfs/vmware/vcf/nfs-mount/bundle/
      
      
4. To retry the download, navigate to Bundle Management > Bundles > Bundle Details in the SDDC Manager UI and click Retry Download.

Additional Information

If you still see the Error follow KB to Cleanup up bundles  : DepotConnectionFailureException: Error downloading bundle [/COMP/SDDC_MANAGER_VCF/bundles/bundle-<TBD>.tar] from dl.broadcom:443

In some upgraded environments(IE 9.0.1 and 5.2.2) the nfs-mount folder will show nginx as the user. This is a known issue and will be fixed in a future version.