Aria Operations for Networks upgrade fails with Insufficient disk space message displayed in Aria Operations for Networks GUI

Products

VCF Operations for Networks

Issue/Introduction

When trying to update Aria Operations for Networks via any method (via GUI or via Aria Suite Lifecycle manager (vRSLCM), upgrade fails with Insufficient disk space message displayed in Aria Operations for Networks GUI or failed disk space check in Aria Suite Lifecycle Manager

Refer to Aria Operations for Networks error screenshot below:

Error: Insufficient disk space.

Refer to Aria Suite Lifecycle manager (vRSLCM) error screenshot below:

Status: Failed
Check: Disk space check on /.
Component / Resource: Disk space check on /.
Result Description: Disk space on / should be at least 12GB for upgrade. Current available space on is ##GB in node.

Running df -h command on Aria Operations for Networks platform appliance shows "/" with 11 G available.
Refer to example output below :

support@vrni-platform-release:~$ df -h
Filesystem                    Size  Used Avail Use% Mounted on
udev                           32G     0   32G   0% /dev
tmpfs                         6.3G   16M  6.3G   1% /run
/dev/mapper/vg-root            29G   17G   11G  63% /
tmpfs                          32G   16K   32G   1% /dev/shm 
tmpfs                         5.0M     0  5.0M   0% /run/lock
/dev/sda1                     1.9G  122M  1.6G   8% /boot
none                           32G     0   32G   0% /run/shm
/dev/mapper/vg-tmp             25G  1.7G   22G   8% /tmp
/dev/mapper/vg-home            30G  8.7G   20G  32% /home
/dev/mapper/vg-var            852G   14G  794G   2% /var
/dev/mapper/vg-var+log         49G   18G   29G  38% /var/log
/dev/mapper/vg-var+log+audit   20G   62M   19G   1% /var/log/audit
tmpfs                         6.3G     0  6.3G   0% /run/user/998
tmpfs                         6.3G     0  6.3G   0% /run/user/115
tmpfs                         6.3G     0  6.3G   0% /run/user/119
tmpfs                         6.3G     0  6.3G   0% /run/user/117
tmpfs                         6.3G     0  6.3G   0% /run/user/120
tmpfs                         6.3G     0  6.3G   0% /run/user/1001
support@vrni-platform-release:~$

Above same command can be run on Aria Operations for Networks deployments running platform in cluster but only on platform node1.

Environment

Aria Operations for Networks 6.12.0
Aria Operations for Networks 6.12.1
Aria Operations for Networks 6.13.0
Aria Operations for Networks 6.14.0

Cause

Aria Operations for Networks upgrade fails because the old bundle did not get auto cleaned up during the previous upgrades hence consuming space on "/" (root) partition.

Resolution

This issue is fixed in Aria Operations for Networks 6.14.0 version. Upgrade done on 6.14.0 and above will not see this issue.

To workaround this, follow below steps to perform a manual clean up of the older extracted bundles:

1. Take a SSH/Putty session to Aria Operations for Networks platform appliance. If you are running Aria Operations for Networks deployment with platform cluster, then take SSH/Putty session on platform Node1

2. Login with username support and run below commands:

support@vrni-platform-release:~$ ub
ubuntu@vrni-platform-release::~$ ls -lrth /usr/share/nginx/upgradebundle
total 8.0K
drwxr-xr-x 3 www-data www-data 4.0K Nov  2 06:13 6.13.0.1717138317
drwxr-xr-x 3 www-data www-data 4.0K Nov  2 06:13 6.12.1.1707992545
drwxr-xr-x 2 www-data www-data 4.0K Nov  2 08:09 6.12.0.1706185032

As seen in output above you should see older extracted bundles in your Aria Operations for Networks appliance.

3. Run below command for manual clean up depending on the older bundle file name you see in output of the command executed in step 2 above.

ubuntu@vrni-platform-release:~$ sudo rm -rf /usr/share/nginx/upgradebundle/<bundlename>
Example commands:
ubuntu@vrni-platform-release:~$ sudo rm -rf /usr/share/nginx/upgradebundle/6.13.0.1717138317/
ubuntu@vrni-platform-release:~$ sudo rm -rf /usr/share/nginx/upgradebundle/6.12.1.1707992545/
ubuntu@vrni-platform-release:~$ sudo rm -rf /usr/share/nginx/upgradebundle/6.12.0.1706185032/

5. To validate if clean/manual removal of bundle is successful run below command.

ubuntu@vrni-platform-release:~$ ls -lrth /usr/share/nginx/upgradebundle
total 0

6. Now run below command to see that Aria Operations for Networks platform appliance shows "/" with more than 14 G.
Refer below command used and output

ubuntu@vrni-platform-release:~$ df -h
Filesystem                    Size  Used Avail Use% Mounted on
udev                           32G     0   32G   0% /dev
tmpfs                         6.3G   16M  6.3G   1% /run
/dev/mapper/vg-root            29G  9.7G   18G  36% /
tmpfs                          32G   16K   32G   1% /dev/shm
tmpfs                         5.0M     0  5.0M   0% /run/lock
/dev/sda1                     1.9G  122M  1.6G   8% /boot
none                           32G     0   32G   0% /run/shm
/dev/mapper/vg-tmp             25G  1.7G   22G   8% /tmp
/dev/mapper/vg-home            30G  8.7G   20G  32% /home
/dev/mapper/vg-var            852G   14G  794G   2% /var
/dev/mapper/vg-var+log         49G   18G   29G  38% /var/log
/dev/mapper/vg-var+log+audit   20G   64M   19G   1% /var/log/audit
tmpfs                         6.3G     0  6.3G   0% /run/user/998
tmpfs                         6.3G     0  6.3G   0% /run/user/115
tmpfs                         6.3G     0  6.3G   0% /run/user/119
tmpfs                         6.3G     0  6.3G   0% /run/user/117
tmpfs                         6.3G     0  6.3G   0% /run/user/120
tmpfs                         6.3G     0  6.3G   0% /run/user/1001
ubuntu@vrni-platform-release:~$
ubuntu@platform1:~$ exit
logout
support@platform1:~$

Once above steps are executed for manual clean up the Aria Operations for Networks upgrade needs to be kickstarted again and should pass the pre check, toast message on UI would not been seen and upgrade proceeds as normal.