DepotConnectionFailureException: Error downloading bundle [/COMP/SDDC_MANAGER_VCF/bundles/bundle-<TBD>.tar] from dl.broadcom:443

Products

VMware SDDC Manager

Issue/Introduction

SDDC Manager - Bundle Manager is failing to download bundles. Download task takes a long time to start.
Unable to download update through SDDC Manager.
The UI displays one of the following specific exceptions depending on the bundle and depot source:

DepotConnectionFailureException: Error downloading bundle [/COMP/SDDC_MANAGER_VCF/bundles/bundle-<TBD>.tar] from dl.broadcom:443

DepotConnectionFailureException: Error downloading bundle [/evo/vmw/bundles/bundle-#####.tar] from depot.vmware.com:443

Reviewing the LCM debug log (/var/log/vmware/vcf/lcm/lcm-debug.log) reveals a Repo Write Failure

DEBUG [vcf_lcm,ba0cd####e8#####1,ed##] [c.v.e.s.l.p.impl.vsan.VsanHealthUtil,http-nio-127.0.0.1-7##0-exec-6] Health query Result: HealthCheckQueryResult(resultId=0####45-d##c-##f3-b7##-83######2e49, status=SUCCESS, result=[])
DEBUG [vcf_lcm,ba0cd####e8#####1,ed##] [c.v.v.l.r.a.c.v.v.HealthCheckController,http-nio-127.0.0.1-7400-exec-6] getHealthCheckStatus result: HealthCheckQueryResult(resultId=0####45-d##c-##f3-b7##-83######2e49, status=SUCCESS, result=[])
ERROR [vcf_lcm,699323############7b7c49c1df,e8##f,bundleId=8b######3-d##d-###4-##9d-4#######9b6e] [c.v.e.s.l.f.b.c.i.BundleFileClientImpl,vac-scheduler-1] Failed to store bundle /nfs/vmware/vcf/nfs-mount/bundle/depot/local/bundles/bundle-251#####8.tar into the repository
ERROR [vcf_lcm,699323############7b7c49c1df,e8af] [c.v.e.s.l.s.i.BundleDownloadExceptionHandlerImpl,vac-scheduler-1] Bundle download failed
com.vmware.evo.sddc.lcm.model.bundle.repo.exception.BundleRepoWriteFailureException: Failed to store bundle(REPO WRITE FAILURE, path = /nfs/vmware/vcf/nfs-mount/bundle/depot/local/bundles/bundle-251#####8.tar
        at com.vmware.evo.sddc.lcm.file.bundle.client.impl.BundleFileClientImpl.storeBundle(BundleFileClientImpl.java:142)

The NFS mount path /nfs/vmware/vcf/nfs-mount shows high or near 100% disk utilization. We can verify the same using below command:
```
df -h
```

Environment

VMware Cloud Foundation 5.2

Cause

Issue is the lack of space on /nfs/vmware/vcf/nfs-mount/ and when the SDDC Manager attempts to write the compressed bundle to the local repository, the underlying filesystem hits its capacity limit. This triggers a REPO WRITE FAILURE, as the BundleFileClientImpl cannot finalize the file transfer to the disk.

Behavior is reproducible via triggering a bundle via GUI and tail-ing the lcm-debug.log

# tail -f /var/log/vmware/vcf/lcm/lcm-debug.log | grep -i "no space left on device"

Can also be listed via
root@sddc_manager [ ~ ]# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 4.0M 0 4.0M 0% /dev tmpfs 7.9G 32K 7.9G 1% /dev/shm tmpfs 3.2G 8.8M 3.2G 1% /run tmpfs 4.0M 0 4.0M 0% /sys/fs/cgroup /dev/mapper/vg_system-lv_root 26G 15G 10G 59% / /dev/mapper/vg_lvm_snapshot-lv_lvm_snapshot 87G 24K 82G 1% /storage/lvm_snapshot /dev/mapper/vg_alt_root-lv_alt_root 26G 24K 25G 1% /storage/alt_root tmpfs 7.9G 40M 7.8G 1% /tmp /dev/mapper/vg_lcmmount-lv_lcmmount 503G 227G 252G 100% /nfs/vmware/vcf/nfs-mount /dev/mapper/vg_data-lv_data3 165G 5.0G 151G 4% /opt/vmware/vcf /dev/sda3 488M 48M 404M 11% /boot /dev/sda2 10M 2.0M 8.1M 20% /boot/efi /dev/mapper/vg_data-lv_data2 63G 39G 21G 66% /var/log/vmware /dev/mapper/vg_data-lv_data1 7.8G 132M 7.3G 2% /data

Resolution

To resolve this issue, free up disk space by removing unused and old bundles from the NFS repository.

SSH to the SDDC Manager VM with vcf user credentials and then switch to root login.
Verify Disk Utilization using below command:
```
df -h
```
Run the following command to verify which specific subdirectory is consuming the most space:
```
du -sh /nfs/vmware/vcf/nfs-mount/*
```
List the contents of the bundle directory to find the IDs of older bundles that are no longer needed:
```
ls -lh /nfs/vmware/vcf/nfs-mount/bundle/depot/local/bundles/
```
Note: The Bundle ID associated with large files you wish to delete (e.g., if the file is bundle-12345.tar, the ID is 12345).
List the files to identify older bundles that are no longer needed. Remove them using below command:
```
/opt/vmware/vcf/lcm/lcm-app/bin/bundle_cleanup.py {bundle id}
```
For multiple bundles, use the same CLI, as listed below
```
python /opt/vmware/vcf/lcm/lcm-app/bin/bundle_cleanup.py "{Bundle id -#####}" "{Bundle id -#####}"
```
Reference: How to clean up old/ installed bundles in SDDC manager using Bundle Cleanup Utility
Run df -h again to confirm that sufficient space has been reclaimed. Log back into the SDDC Manager UI and retry the bundle download process.