Error: Disk space on "/" should be at least 12GB during upgrade

Products

VCF Operations for Networks

Issue/Introduction

Note: VCF Operations for Networks was formerly named Aria Operations for Networks (AON), and prior to that was named vRealize Network Insight (vRNI).

In the Aria Operations for Networks GUI the uploading of new upgrade OVA version fails displaying the percent uploaded prior to failure.
Uploading of 6.14.3 upgrade bundle fails.
When trying to update Aria Operations for Networks via any method (via GUI or via Aria Suite Lifecycle manager (vRSLCM), upgrade fails with Insufficient disk space message displayed in Aria Operations for Networks GUI or failed disk space check in Aria Suite Lifecycle Manager.
The following error appears: Disk space on / should be at least 12GB for upgrade. Current available space on / is ####GB in node ####.
Running df -h / shows the root partition has less than 12GB of available space.

Refer to Aria Operations for Networks error screenshot below:

Error: Insufficient disk space.

Refer to Aria Suite Lifecycle manager (vRSLCM) error screenshot below:

Status: Failed
Check: Disk space check on /.
Component / Resource: Disk space check on /.
Result Description: Disk space on / should be at least 12GB for upgrade. Current available space on is ##GB in node.

Running df -h command on Aria Operations for Networks platform appliance shows "/" with 11 G available.
Refer to example output below :

support@vrni-platform-release:~$ df -h
Filesystem                    Size  Used Avail Use% Mounted on
udev                           32G     0   32G   0% /dev
tmpfs                         6.3G   16M  6.3G   1% /run
/dev/mapper/vg-root            29G   17G   11G  63% /
tmpfs                          32G   16K   32G   1% /dev/shm 
tmpfs                         5.0M     0  5.0M   0% /run/lock
/dev/sda1                     1.9G  122M  1.6G   8% /boot
none                           32G     0   32G   0% /run/shm
/dev/mapper/vg-tmp             25G  1.7G   22G   8% /tmp
/dev/mapper/vg-home            30G  8.7G   20G  32% /home
/dev/mapper/vg-var            852G   14G  794G   2% /var
/dev/mapper/vg-var+log         49G   18G   29G  38% /var/log
/dev/mapper/vg-var+log+audit   20G   62M   19G   1% /var/log/audit
tmpfs                         6.3G     0  6.3G   0% /run/user/998
tmpfs                         6.3G     0  6.3G   0% /run/user/115
tmpfs                         6.3G     0  6.3G   0% /run/user/119
tmpfs                         6.3G     0  6.3G   0% /run/user/117
tmpfs                         6.3G     0  6.3G   0% /run/user/120
tmpfs                         6.3G     0  6.3G   0% /run/user/1001
support@vrni-platform-release:~$

Above same command can be run on Aria Operations for Networks deployments running platform in cluster but only on platform node1.

Environment

VCF Operations for Networks 6.13.0
VCF Operations for Networks 6.14.x
VCF Operations for Networks 9.0.2

Cause

The root (/) partition lacks the required 12GB of free space. This is typically caused by old support bundles or the extracted upgrade bundle of the currently active version remaining in the system.

Resolution

Deleting the current version's upgrade bundle in /usr/share/nginx/upgradebundle/ is a valid workaround when disk space is below 12GB, as binaries are already staged.

To workaround this, follow below steps to perform a manual cleanup of the older extracted bundles:

Take a SSH/Putty session to Aria Operations for Networks platform appliance. If you are running Aria Operations for Networks deployment with platform cluster, then take SSH/Putty session on platform Node1.
Login with username support and run below commands:

support@vrni-platform-release:~$ ub
ubuntu@vrni-platform-release::~$ ls -lrth /usr/share/nginx/upgradebundle
total 16.0K
drwxr-xr-x 2 www-data www-data 4.0K Nov 2 08:09 6.14.0.1725688792
drwxr-xr-x 3 www-data www-data 4.0K Nov 2 08:09 6.13.0.1717138317
drwxr-xr-x 3 www-data www-data 4.0K Sep 14 20:17 9.0.2.25119537

As seen in the output above, older extracted bundles may be present in the Aria Operations for Networks appliance.
Run the below command for manual cleanup depending on the older bundle file name you see in the output of the command executed in step 2 above.

ubuntu@vrni-platform-release:~$ sudo rm -rf /usr/share/nginx/upgradebundle/<older-bundlename>

To validate if manual removal of the older bundles is successful, run the below command:

ubuntu@platform1:~$ ls -lrth /usr/share/nginx/upgradebundle/
total 0
ubuntu@platform1:~$

Run below command to see that Aria Operations for Networks platform appliance shows "/" with more than 14 G.

Refer below command used and output:

ubuntu@platform1:~$ df -h
Filesystem                    Size  Used Avail Use% Mounted on
tmpfs                         6.3G  1.3M  6.3G   1% /run
/dev/mapper/vg-root            29G   13G   15G  46% /
tmpfs                          32G   64K   32G   1% /dev/shm
tmpfs                         5.0M     0  5.0M   0% /run/lock
/dev/sda1                     1.8G  213M  1.5G  13% /boot
none                           32G     0   32G   0% /run/shm
/dev/mapper/vg-tmp             25G  8.7M   24G   1% /tmp
/dev/mapper/vg-home            30G   12G   17G  41% /home
/dev/mapper/vg-var            1.9T   19G  1.8T   2% /var
/dev/mapper/vg-var+log         49G   13G   34G  27% /var/log
/dev/mapper/vg-var+log+audit   20G   19M   19G   1% /var/log/audit
tmpfs                         6.3G     0  6.3G   0% /run/user/997
tmpfs                         6.3G     0  6.3G   0% /run/user/1000
tmpfs                         6.3G     0  6.3G   0% /run/user/996
tmpfs                         6.3G     0  6.3G   0% /run/user/993
tmpfs                         6.3G     0  6.3G   0% /run/user/995
tmpfs                         6.3G     0  6.3G   0% /run/user/1002
ubuntu@platform1:~$

If Disc space still shows less than 12 G then check for older/obsolete system files, including:

Multiple old versions of Python 3 binaries (e.g., 3.5, 3.6) stored in `/usr/local/lib`.
Residual UI support bundles in `/ui-support-bundles`.
To resolve this issue, you must manually reclaim space on the root partition.

7. Remove Old Support Bundles

8. Check for and delete any old support bundles that are no longer needed, steps below

9. Execute below command to delete older bundles

cd /ui-support-bundles
ls -al
sudo rm -f sb.*.tar.gz

Note: The above command cannot be undone. Ensure you verify every parameter before running.

10. Remove Obsolete Python Binaries:
Identify which Python versions are consuming space. In some environments, versions 3.5 and 3.6 may remain after system upgrades.
Review this command before running it as follows:

du -xhd 1 /usr/local/lib | sort -h

11. Now execute below command to clean up the older python libraries

sudo rm -rf /usr/local/lib/python3.5
sudo rm -rf /usr/local/lib/python3.6

Note: Above command cannot undone, hence, verify every parameter before running.

12. Confirm that the available space on the root partition is now 12GB or greater. Execute below command.

df -h

13. Once above steps are executed for manual clean up and disc space is reclaimed the Aria Operations for Networks upgrade needs to be kickstarted again and should pass the pre check, toast message on UI would not been seen and upgrade proceeds as normal.

14. If you are using VMware Aria Suite Lifecycle, then navigate to its GUI re-run the failed upgrade precheck.

If the issue persists, contact Broadcom Support. For software downloads, visit the Broadcom Support Portal.