Aria Operations for Networks upgrade to GA version(s) and patch version(s) fails with Insufficient disk space message displayed in Aria Operations for Networks GUI

Products

• VCF Operations for Networks

Issue/Introduction

• In the Aria Operations for Networks GUI the uploading of new upgrade OVA version fails  displaying the percent uploaded prior to failure
• When trying to update Aria Operations for Networks via any method (via GUI or via Aria Suite Lifecycle manager (vRSLCM), upgrade fails with Insufficient disk space message displayed in Aria Operations for Networks GUI or failed disk space check in Aria Suite Lifecycle Manager.
  
  This Knowledge base article also applies for upgrade to GA version and patch(s)

Refer to Aria Operations for Networks error screenshot below:

Error: Insufficient disk space.

 

• Refer to Aria Suite Lifecycle manager (vRSLCM) error screenshot below:
  
  Status: Failed
  Check: Disk space check on /.
  Component / Resource: Disk space check on /.
  Result Description: Disk space on / should be at least 12GB for upgrade. Current available space on  is ##GB in node.

 

• Running df -h command on Aria Operations for Networks platform appliance shows "/" with 11 G available. 
  Refer to example output below :
  
  

support@vrni-platform-release:~$ df -h
Filesystem                    Size  Used Avail Use% Mounted on
udev                           32G     0   32G   0% /dev
tmpfs                         6.3G   16M  6.3G   1% /run
/dev/mapper/vg-root            29G   17G   11G  63% /
tmpfs                          32G   16K   32G   1% /dev/shm 
tmpfs                         5.0M     0  5.0M   0% /run/lock
/dev/sda1                     1.9G  122M  1.6G   8% /boot
none                           32G     0   32G   0% /run/shm
/dev/mapper/vg-tmp             25G  1.7G   22G   8% /tmp
/dev/mapper/vg-home            30G  8.7G   20G  32% /home
/dev/mapper/vg-var            852G   14G  794G   2% /var
/dev/mapper/vg-var+log         49G   18G   29G  38% /var/log
/dev/mapper/vg-var+log+audit   20G   62M   19G   1% /var/log/audit
tmpfs                         6.3G     0  6.3G   0% /run/user/998
tmpfs                         6.3G     0  6.3G   0% /run/user/115
tmpfs                         6.3G     0  6.3G   0% /run/user/119
tmpfs                         6.3G     0  6.3G   0% /run/user/117
tmpfs                         6.3G     0  6.3G   0% /run/user/120
tmpfs                         6.3G     0  6.3G   0% /run/user/1001
support@vrni-platform-release:~$

• Above same command can be run on Aria Operations for Networks deployments running platform in cluster but only on platform node1.
  
  
  

Environment

VCF Operations for Networks 6.13.0
VCF Operations for Networks 6.14.x
VMware Aria Suite Lifecycle 8.18.x

Cause

Aria Operations for Networks upgrade fails because the old bundle did not get auto cleaned up during the previous upgrades hence consuming space  on "/" (root) partition.

Resolution

This issue is fixed in Aria Operations for Networks 6.14.0 version. Upgrade done on 6.14.0 and above will not see this issue.

To workaround this, follow below steps to perform a manual cleanup of the older extracted bundles:

1. Take a SSH/Putty session to Aria Operations for Networks platform appliance. If you are running Aria Operations for Networks deployment with platform cluster, then take SSH/Putty session on platform Node1

2. Login with username support and run below commands:

support@vrni-platform-release:~$ ub
ubuntu@vrni-platform-release::~$ ls -lrth /usr/share/nginx/upgradebundle
total 16.0K
drwxr-xr-x 2 www-data www-data 4.0K Nov  2 08:09 6.14.0.1725688792
drwxr-xr-x 3 www-data www-data 4.0K Nov  2 08:09 6.13.0.1717138317
drwxr-xr-x 3 www-data www-data 4.0K Nov  2 06:13 6.12.1.1707992545
drwxr-xr-x 2 www-data www-data 4.0K Nov  2 08:09 6.12.0.1706185032

As seen in the output above, older extracted bundles may be present in the Aria Operations for Networks appliance.

3. Run the below command for manual cleanup depending on the older bundle file name you see in the output of the command executed in step 2 above.

ubuntu@vrni-platform-release:~$ sudo rm -rf /usr/share/nginx/upgradebundle/<older-bundlename>
Example commands:
ubuntu@vrni-platform-release:~$ sudo rm -rf /usr/share/nginx/upgradebundle/6.13.0.1717138317
ubuntu@vrni-platform-release:~$ sudo rm -rf /usr/share/nginx/upgradebundle/ 6.12.1.1707992545
ubuntu@vrni-platform-release:~$ sudo rm -rf /usr/share/nginx/upgradebundle/ 6.12.0.1706185032

Note: By design, VCF Operations for Networks retains the latest extracted upgrade bundle after an upgrade, while older extracted bundles are removed. If older bundle directories are still present on the appliance, remove only the older bundle directories identified in step 2. Do not remove the latest retained bundle for the version currently running or being upgraded to.

4. To validate if manual removal of the older bundles is successful, run the below command.

ubuntu@vrni-platform-release:~$ ls -lrth /usr/share/nginx/upgradebundle
drwxr-xr-x 2 www-data www-data 4.0K Nov  2 08:09 6.14.0.1725688792

5. Now run below command to see that Aria Operations for Networks platform appliance shows "/" with more than 14 G.
 
Refer below command used and output:

ubuntu@vrni-platform-release:~$ df -h
Filesystem                    Size  Used Avail Use% Mounted on
udev                           32G     0   32G   0% /dev
tmpfs                         6.3G   16M  6.3G   1% /run
/dev/mapper/vg-root            29G  9.7G   18G  36% /
tmpfs                          32G   16K   32G   1% /dev/shm
tmpfs                         5.0M     0  5.0M   0% /run/lock
/dev/sda1                     1.9G  122M  1.6G   8% /boot
none                           32G     0   32G   0% /run/shm
/dev/mapper/vg-tmp             25G  1.7G   22G   8% /tmp
/dev/mapper/vg-home            30G  8.7G   20G  32% /home
/dev/mapper/vg-var            852G   14G  794G   2% /var
/dev/mapper/vg-var+log         49G   18G   29G  38% /var/log
/dev/mapper/vg-var+log+audit   20G   64M   19G   1% /var/log/audit
tmpfs                         6.3G     0  6.3G   0% /run/user/998
tmpfs                         6.3G     0  6.3G   0% /run/user/115
tmpfs                         6.3G     0  6.3G   0% /run/user/119
tmpfs                         6.3G     0  6.3G   0% /run/user/117
tmpfs                         6.3G     0  6.3G   0% /run/user/120
tmpfs                         6.3G     0  6.3G   0% /run/user/1001
ubuntu@vrni-platform-release:~$
ubuntu@platform1:~$ exit
logout
support@platform1:~$

 

6. If disc space still shows less than 12 G then check for older/obsolete system files, including:

1. Multiple old versions of Python 3 binaries (e.g., 3.5, 3.6, 3.8) stored in `/usr/local/lib`.
2. Residual UI support bundles in `/ui-support-bundles`.
3. To resolve this issue, you must manually reclaim space on the root partition. 

7.  Remove Old Support Bundles
8. Check for and delete any old support bundles that are no longer needed, steps below

9. Execute below command to delete older bundles 

cd /ui-support-bundles
ls -al
sudo rm -f sb.*.tar.gz

Note: The above command cannot be undone. Ensure you verify every parameter before running.

10. Remove Obsolete Python Binaries:
      Identify which Python versions are consuming space. In some environments, versions 3.5, 3.6, and 3.8 may remain after system updates.
      Review this command before running it as follows:

du -xhd 1 /usr/local/lib | sort -h

11. Now execute below command to clean up the older python libraries

sudo rm -rf /usr/local/lib/python3.5
sudo rm -rf /usr/local/lib/python3.6
sudo rm -rf /usr/local/lib/python3.8

Note: Above command cannot undone, hence, verify every parameter before running.

12. Confirm that the available space on the root partition is now 12GB or greater. Execute below command.

df -h

13. Once above steps are executed for manual clean up and disc space is reclaimed  the Aria Operations for Networks upgrade needs to be kickstarted again and should pass the pre check, toast message on UI would not been seen and upgrade proceeds as normal.

14. If you are using VMware Aria Suite Lifecycle, then navigate to its  GUI re-run the failed upgrade precheck.