Upgrade error Boot partition size check in Aria Operations for Networks Appliances before upgrades

Products

• VMware Aria Operations for Networks

Issue/Introduction

Boot Partition Migration is needed for Aria Operations for Networks Appliances before upgrades to avoid upgrade failures or upgrade struck issues in cases where the boot partition size is too small.

The boot partition size on Aria Operations for Networks (vRNI) setups which started before 6.0 was only 183 MB.

This issue started surfacing for these customers when the setup ran out of boot partition space during upgrades.

Aria Operations for Networks deployments which started with 6.0 or later have a boot size of 1.8GB as default on the appliance hence these appliance were not impacted.

Symptoms:

1. Upgrade fails with below error message showing precheck failure on upgrade GUI.
   
2. Executing df-h commands shows boot partition size is significantly less, for example: 179 M as in below output.
   

   ubuntu@aria-networks-platform:~$ ./run_all.sh df -h
   This is not a cluster environment. Running the command on localhost only...
   Filesystem                    Size  Used Avail Use% Mounted on
   udev                           32G     0   32G   0% /dev
   tmpfs                         6.3G  2.6M  6.3G   1% /run
   /dev/mapper/vg-root            29G   22G  5.1G  82% /
   tmpfs                          32G  8.0K   32G   1% /dev/shm
   tmpfs                         5.0M     0  5.0M   0% /run/lock
   /dev/sda1                     179M  133M   33M  81% /boot
   none                           32G     0   32G   0% /run/shm
   /dev/mapper/vg-tmp             25G  1.3G   23G   6% /tmp
   /dev/mapper/vg-home            25G   13G   11G  55% /home
   /dev/mapper/vg-var            602G   55G  517G  10% /var
   /dev/mapper/vg-var+log         49G   29G     0  20% /var/log
   /dev/mapper/vg-var+log+audit   20G   19M   19G   1% /var/log/audit
   tmpfs                         6.3G     0  6.3G   0% /run/user/115
   tmpfs                         6.3G     0  6.3G   0% /run/user/999
   tmpfs                         6.3G     0  6.3G   0% /run/user/116
   tmpfs                         6.3G     0  6.3G   0% /run/user/119
   tmpfs                         6.3G     0  6.3G   0% /run/user/1001

3. Upgrade history for appliance in this example shows an older version:
   

   ubuntu@aria-networks-platform:~$ sudo cat /home/ubuntu/build-target/deployment/patch.txt
   [Sep 07 2018 18:22:42-015] 3.9.0.1536339775
   [Sep 07 2018 18:33:22-025] 3.9.0.1536339775
   [12/14/2018 16:23:43] Chrome71Support-201812051800 Success
   [03/18/2019 23:24:33] 4.0.0.1545292702 Success
   [Sep 10 2019 19:13:24] 4.2.0.1562947515 Success
   [Aug 03 2021 21:05:27] 5.1.0.1578493419 Success
   [Aug 05 2021 00:49:47] 5.3.0.1593829198 Success
   [Aug 05 2021 15:36:11] 6.0.0.1601964323 Success
   [Aug 05 2021 17:40:25] 6.2.0.1617345572 Success
   [Aug 05 2021 21:19:49] 6.3.0.1625072760 Success
   [May 10 2023 16:40:27] 6.5.1.1645037705 Success
   [May 13 2023 00:04:50] 6.6.0.1648892975 Success
   [May 15 2023 16:15:31] 6.8.0.1666364233 Success
   [May 15 2023 17:51:07] 6.9.0.1673888786 Success
   [May 15 2023 23:20:27] 6.10.0.1680883302 Success
   [Jun 08 2023 15:18:36] 6.10.0.1685358321 (6.10.0.P1)
   [Jun 08 2023 15:23:33] 6.10.0.1685358321 Success
   [Aug 29 2023 23:47:50] 6.11.0.1692527086 Success

Environment

VMware vRealize Network Insight version 3.x
VMware vRealize Network Insight version 4.x
VMware vRealize Network Insight version 5.x
VMware vRealize Network Insight version 6.x
Aria Operations for Networks 6.10
Aria Operations for Networks 6.11

 

Cause

When you update your system, it needs to make some changes to system files stored in a place called the ‘boot’ directory. If there isn’t enough space in this directory, you might see an error message saying ‘boot is full and no space left on device’. This can happen if there are leftover files from old updates, or if the new update is larger than usual. Each update or upgrade might need a bit more space than the last one.

During an operating system upgrade, the kernel compresses and creates compressed files in the boot directory. If the boot space is insufficient to accommodate all kernel sizes during runtime, an error message indicating ‘boot is full and no space left on device’ will be displayed.
This situation can arise due to several factors, such as the presence of stale entries from old kernels in the boot space or an increase in the size of the latest kernel. It’s important to note that kernel size tends to increase with each operating system update and upgrade.

 

Resolution

Aria Operations for Networks deployments created before v6.0 come with a 200MB boot partition by default.

Since then it has been increased by default to 2GB for newer deployments to accommodate present requirements.

To resolve this issue, expansion of the boot partition for these older deployments is needed or upgrade might fail with the symptoms mentioned above due to increased space needs.

Contact Broadcom Support by opening support case to obtain assistance with expanding the boot partition.