NSX-T VIBs fail to install/upgrade, due to insufficient space in bootbank on ESXi host.
Article ID: 324231
Updated On:
Products
Issue/Introduction
Symptoms:
- When attempting to install NSX-T on ESXi host, it may fail.
- ESXi host logs (esxupdate.log) display message(s) similar to:
Environment
2.x
3.x
4.x
Cause
This issue occurs when the bootbank/altbootbank runs out of space on the ESXi host. There is no supported procedure to increase the size of either bootbank on the ESXi host.
Some ESXi images provided by third-party vendors may include VIBs which are not in use and can be relatively large in size. This can result in insufficient space in bootbank/altbootbank when installing/upgrading any VIBs.
Resolution
This issue is resolved in ESXi 8.0 which introduced a larger bootbank for both upgraded and newly installed ESXi hosts.
Workaround:
To work around this issue, free the additional space on the bootbank by uninstalling non-required VIBs on the ESXi host using the 'esxcli software vib remove -n <vib-name>' command.
1.) Check the free space in bootbank / altbootbank:
ls -l ... lrwxrwxrwx 1 root root 49 Aug 9 19:06 altbootbank -> /vmfs/volumes/########-########-####-############ <ALTBootbankUUID> drwxr-xr-x 1 root root 512 Aug 9 19:04 bin lrwxrwxrwx 1 root root 49 Aug 9 19:06 bootbank -> /vmfs/volumes/########-########-####-############ <BootbankUUID>
To see how much space is used and free:
df -h Filesystem Size Used Available Use% Mounted on VMFS-6 383.8G 125.0G 258.7G 33% /vmfs/volumes/datastore1 vfat 285.8M 172.9M 112.9M 60% /vmfs/volumes/########-########-####-############
vfat 249.7M 149.6M 100.1M 60% /vmfs/volumes/########-########-####-############ <ALTBootbankUUID> -> partitions used for altbootbank, see above vfat 249.7M 150.5M 99.2M 60% /vmfs/volumes/########-########-####-############ <BootbankUUID> -> partitions used for bootbank, see above
2.) Check which partition (bootbank or altbootbank) contain the VIBs last updated by running the below commands:
grep updated /bootbank/boot.cfg
updated=8
grep updated /altbootbank/boot.cfg
updated=7
If bootbank has the higher value for “updated”, as in the example above, then that partition has the latest update to the VIBs. Otherwise altbootbank would have the latest update to the VIBs.
3.) NSX software space requirements, note reduced sizing from 3.0
NSX-T 2.4.0 - VMware ESXi 6.5 - 32.17 MB
NSX-T 2.4.0 - VMware ESXi 6.7 - 32.55 MB
NSX-T 2.5.0 - VMware ESXi 6.5 - 37.24 MB
NSX-T 2.5.0 - VMware ESXi 6.7 - 37.64 MB
NSX-T 3.0.0 - VMware ESXi 6.5 - 29.16 MB
NSX-T 3.0.0 - VMware ESXi 6.7 - 29.42 MB
NSX-T 3.0.0 - VMware ESXi 7.0 - 29.43 MB
As well as this space requirement for the NSX VIBs, ESX install/upgrade framework has the following requirements:
For an NSX-T upgrade, an additional 20MB free space buffer is required in the partition with the highest updated value (check step 2 above to find the partition with highest updated value).
For a new install of NSX-T, an additional 10MB free space buffer is required in the partition with highest updated value.
4.) To determine the currently installed VIBs in bootbank/altbootbank partitions run the following command:
ls -lSrh /bootbank/ ls -lSrh /altboobank/
Note: If the ESXi server was installed using a software image provided by a third party, you need to contact the third party and work with them to identify the VIBs which can be safely removed.
5.) Once the VIBs to be removed have been determined, remove the unused VIBs using command:
esxcli software vib remove -n <vib-name>
If any issues are encountered during the above process, file a support request with Broadcom support and quote this Knowledge base article (324231) in the problem description.
6.) Once the required VIBs have been removed the host will require a reboot to show the newly allocated space on the running configuration of bootbank. Once space is verified proceed as indicated in Step 3 with install or upgrade.
Feedback
