DepotConnectionFailureException: Error downloading bundle [/COMP/SDDC_MANAGER_VCF/bundles/bundle-<TBD>.tar] from dl.broadcom:443

Products

• VMware SDDC Manager

Issue/Introduction

SDDC Manager - Bundle Manager is failing to download bundles.
Download task takes a long time to start, and fails eventually with:

DepotConnectionFailureException: Error downloading bundle [/COMP/SDDC_MANAGER_VCF/bundles/bundle-<TBD>.tar] from dl.broadcom:443

This issue is happening even after the following knowledge articles have been applied:
- https://knowledge.broadcom.com/external/article/389871  - application-prod.properties manual entries are correct/ confirmed

- https://knowledge.broadcom.com/external/article/392923/sddc-manager-fail-to-download-bundles-w.html - /nfs/vmware/vcf/nfs-mount permissions issue, that generates the same error message "DepotConnectionFailureException: Error downloading bundle..."

Environment

VCF 5.x

Cause

Issue is the lack of space on /nfs/vmware/vcf/nfs-mount/

Behavior is reproducible via triggering a bundle via GUI and tail-ing the lcm-debug.log
# tail -f /var/log/vmware/vcf/lcm/lcm-debug.log | grep -i "no space left on device"

Can also be listed via 
root@sddc_manager [ ~ ]# df -h
Filesystem                                   Size  Used Avail Use% Mounted on
devtmpfs                                     4.0M     0  4.0M   0% /dev
tmpfs                                        7.9G   32K  7.9G   1% /dev/shm
tmpfs                                        3.2G  8.8M  3.2G   1% /run
tmpfs                                        4.0M     0  4.0M   0% /sys/fs/cgroup
/dev/mapper/vg_system-lv_root                 26G   15G   10G  59% /
/dev/mapper/vg_lvm_snapshot-lv_lvm_snapshot   87G   24K   82G   1% /storage/lvm_snapshot
/dev/mapper/vg_alt_root-lv_alt_root           26G   24K   25G   1% /storage/alt_root
tmpfs                                        7.9G   40M  7.8G   1% /tmp
/dev/mapper/vg_lcmmount-lv_lcmmount          503G  227G  252G 100% /nfs/vmware/vcf/nfs-mount
/dev/mapper/vg_data-lv_data3                 165G  5.0G  151G   4% /opt/vmware/vcf
/dev/sda3                                    488M   48M  404M  11% /boot
/dev/sda2                                     10M  2.0M  8.1M  20% /boot/efi
/dev/mapper/vg_data-lv_data2                  63G   39G   21G  66% /var/log/vmware
/dev/mapper/vg_data-lv_data1                 7.8G  132M  7.3G   2% /data

Resolution

Cleanup up bundles via https://knowledge.broadcom.com/external/article/312173/how-to-bundle-cleanup-utility.html 
# python /opt/vmware/vcf/lcm/lcm-app/bin/bundle_cleanup.py <bundle id>

For multiple bundles, use the same CLI, as listed below
# python /opt/vmware/vcf/lcm/lcm-app/bin/bundle_cleanup.py "yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy" "zzzzzzzz-zzzz-zzzz-zzzz-zzzzzzzzzzzz"

Find the Bundle ID information from the link VMware Cloud Foundation - Upgrade Bundle Details

Can also be caused by the default backup share being created on the same datastore that the bundle download for NSX backups through SDDC.  Older backups can be removed from the command line using rm-r