Upgrade of ESXi hosts to 8.0 fails with an error about missing checksums on their payloads

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

The following errors might be encountered during installation or upgrades of ESXi hosts to 8.0:

The baseline status reports to be Incompatible after running check compliance and below error is observed:
The following VIB(s) on the host or in the chosen baseline(s) do not have the required checksums on their payloads: {VIB names}
This will prevent VIB security verification and secure boot from functioning properly. Please remove these VIBs and check with your vendor for a replacement of these VIBs.
ProfileValidationError is seen when attempting install a VIB or upgrade ESXi host via command line:
In ImageProfile (Updated) {Image Profile Name}, the payload(s) in VIB {VIB name} does not have sha-256 gunzip checksum. This will prevent VIB security verification and secure boot from functioning properly. Please remove this VIB or please check with your vendor for a replacement of this VIB. Please refer to the log file for more details.
Installing the ESXi host fails with below error message:
<MISSING_GUNZIP_CHECKSUM_VIB_ERRORS:
Found=[{VIB names} Expected=[]
These VIB(s) do not have the required sha-256 gunzip checksum for their payloads. This will prevent VIB security verification and secure boot from functioning properly. Please remove these VIBs and check with your vendor for a replacement of these VIBs.>

Environment

VMware vSphere ESXi 8.0.x

Cause

If there are VIBs present on ESXi that do not have the “sha-256” checksum-type and “gunzip” verify-process pair in the VIB metadata, an upgrade to ESXi 8.0 will fail with an error message identifying the VIBs that prevented the upgrade.

Resolution

While the problematic VIBs are part of the host, the upgrade/installation cannot proceed. Thus, one of the following can be done:

Verify with the vendors of the affected VIBs whether there are any updates available for them. If updates are available, please proceed to update these VIBs (utilizing either a VUM baseline or the esxcli commands) and attempt the upgrade operation again.
If the problematic VIBs are not in use or no longer needed, remove them using esxcli command and reattempt the upgrade operation:
- esxcli software vib remove –-vibname <vibname1> –-vibname <vibname2> –-vibname <vibname3> …”

If the following VIBs are installed on the host, the necessary customer action (for upgrading to version 8.0) is detailed below:

VMware-perccli:
- Dell has released newer version of perccli utility for ESXi 7.0 with different name “vmware-perccli64”.
- Customer needs to remove old VIB and install newer one before upgrading.
VMware-storcli
- Broadcom has released newer version of storcli for ESXi 7.0 with different name “vmware-storcli64”.
- Customer needs to remove old VIB and install newer one before upgrading.
emulex-esx-elxmgmt
- Broadcom has newer version of this package for ESXi 7.0 with different name “Broadcom-ELX-esxcli-elxmmt”.
- Customer needs to remove old VIB and install newer one before upgrading.
Avaya: Some Avaya VIBs are not supported anymore, hence needs to be removed for upgrade, this includes:
- asavp-alarming
- avaya-licensing
- avaya-harden
- avaya-watchd
- avp-alarming
New Avaya product offering uses esxcli software profile install -p $<ROFILE>. Profile install instead of profile update to remove previous VIBs and installs the new ones. Following VIBs are being developed with HEXDK version: vmware-esx-hexdk-devtools-7.0.0-1.0.15843807.x86_64, so new 7.0 version will be available:
- avaya-tools
- avaya-easg
- watchd-files
- native-cpld
- cplduser
Dell sas-raid_boss-cli_6.x, SuperMicro TAS:
- Not supported on 8.0
- Remove VIB in order to upgrade to 8.0
NetAppNasPlugin:
- ESXi patch 6.7 is compatible with NetAppNasPlugin 1.1.2-3 which is not supported for upgrade to 8.0.
- NetApp has newer version of this package from ESXi 7.x (NetAppNasPlugin.2.0-15.vib)
- Both methods are viable options when upgrading to version 8.0.
  - Upgrade ESXi =>7.0.1 version followed by updating NetAppNasPlugin to 2.0-15 before 8.0 upgrade.
  - Remove older NetAppNasPlugin vib before upgrade to ESXi8.0. NetAppNasPlugin 2.0-15 vib can be installed after the upgrade.
Nutanix :
- Upgrade the Nutanix VIB before the upgrade.
- VIB Nutanix_bootbank_nfs-vaai-plugin_2.0-5dfbc550
- Try to upgrade the ESXi from "OneClick" Nutanix's Appliance.
- Reach out to Nutanix Vendor to get VIB updated before the ESXi upgrade.

Note: The error in Lifecycle Manager might not contain the offending vibs, and instead show {precheckError}. In these cases, you can determine the problematic vibs from the /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log file:

YYYY-MM-DDTHH:MM:SS.MSZ info vmware-vum-server[07580] [Originator@6876 sub=HostUpgradeScanner] [scannerImpl 661] Precheck script test result: '
ERROR', test 'MISSING_GUNZIP_CHECKSUM_VIBS', expected '', found 'LSI_bootbank_vmware-perccli-007.0318.0000.0000_007.0318.0000.0000-01' and errortype is 42

YYYY-MM-DDTHH:MM:SS.MSZ info vmware-vum-server[07580] [Originator@6876 sub=HostUpgradeScanner] [scannerImpl 1749] (vmodl.LocalizableMessage) [
-->    (vmodl.LocalizableMessage) {
-->       key = "com.vmware.vcIntegrity.HostUpgrade.MissingGunzipChecksumVibs",
-->       arg = (vmodl.KeyAnyValue) [
-->          (vmodl.KeyAnyValue) {
-->             key = "found",
-->             value = "LSI_bootbank_vmware-perccli-007.0318.0000.0000_007.0318.0000.0000-01"
-->          }
-->       ],
-->       message = <unset>
-->    }
--> ]

Additional Information

This issue is validated using VMware Aria Operations User Guide (8.18)

The check is as follows:

Product: vCenter
Log File: vmware-vum-server.log
Log Expression Check "ERROR" AND "MISSING_GUNZIP_CHECKSUM_VIBS"