DepotConnectionFailureException: Error downloading bundle [/COMP/SDDC_MANAGER_VCF/bundles/bundle-<TBD>.tar] from dl.broadcom:443

Products

• VMware SDDC Manager

Issue/Introduction

SDDC Manager - Bundle Manager is failing to download bundles.
Download task takes a long time to start, and fails eventually with:

DepotConnectionFailureException: Error downloading bundle [/COMP/SDDC_MANAGER_VCF/bundles/bundle-<TBD>.tar] from dl.broadcom:443

This issue is happening even after the following knowledge articles have been applied:
- https://knowledge.broadcom.com/external/article/389871  - application-prod.properties manual entries are correct/ confirmed

- https://knowledge.broadcom.com/external/article/392923/sddc-manager-fail-to-download-bundles-w.html - /nfs/vmware/vcf/nfs-mount permissions issue, that generates the same error message "DepotConnectionFailureException: Error downloading bundle..."

Environment

VCF 5.x

Cause

Issue is the lack of space on /nfs/vmware/vcf/nfs-mount/

Behavior is reproducible via triggering a bundle via GUI and tail-ing the lcm-debug.log
# tail -f /var/log/vmware/vcf/lcm/lcm-debug.log | grep -i "no space left on device"

Can also be listed via 
root@sddc_manager [ ~ ]# df -h
Filesystem                                   Size  Used Avail Use% Mounted on
devtmpfs                                     4.0M     0  4.0M   0% /dev
tmpfs                                        7.9G   32K  7.9G   1% /dev/shm
tmpfs                                        3.2G  8.8M  3.2G   1% /run
tmpfs                                        4.0M     0  4.0M   0% /sys/fs/cgroup
/dev/mapper/vg_system-lv_root                 26G   15G   10G  59% /
/dev/mapper/vg_lvm_snapshot-lv_lvm_snapshot   87G   24K   82G   1% /storage/lvm_snapshot
/dev/mapper/vg_alt_root-lv_alt_root           26G   24K   25G   1% /storage/alt_root
tmpfs                                        7.9G   40M  7.8G   1% /tmp
/dev/mapper/vg_lcmmount-lv_lcmmount          503G  227G  252G 100% /nfs/vmware/vcf/nfs-mount
/dev/mapper/vg_data-lv_data3                 165G  5.0G  151G   4% /opt/vmware/vcf
/dev/sda3                                    488M   48M  404M  11% /boot
/dev/sda2                                     10M  2.0M  8.1M  20% /boot/efi
/dev/mapper/vg_data-lv_data2                  63G   39G   21G  66% /var/log/vmware
/dev/mapper/vg_data-lv_data1                 7.8G  132M  7.3G   2% /data

Resolution

Cleanup up bundles via https://knowledge.broadcom.com/external/article/312173/how-to-bundle-cleanup-utility.html 
# python /opt/vmware/vcf/lcm/lcm-app/bin/bundle_cleanup.py <bundle id>

For multiple bundles, use the same CLI, as listed below
# python /opt/vmware/vcf/lcm/lcm-app/bin/bundle_cleanup.py "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" "yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy" "zzzzzzzz-zzzz-zzzz-zzzz-zzzzzzzzzzzz"

Find the Bundle ID information from the link VMware Cloud Foundation - Upgrade Bundle Details