Upgrade Precheck in SDDC Manager fails for ESXi at Upgrade Bundle Validation

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:
When running the Upgrade Precheck in SDDC Manager, the precheck fails for ESXi Hosts checks at the Upgrade bundle validation.

The error description reports entries similar to:

Bundle validation for bundle of type HOST with upgrade version <version> failed due to error
Failed uploading the update/upgrade patch files to VUM, performing compliance checks on the cluster
Failed scanning hosts: [ <list_of_esxi_hosts> ] for the baseline groups associated with the cluster

On the vCenter UI, we see errors reported in the Tasks for the cluster, similar to:

There are errors during the scan operation. Check the events and logs for details.

Environment

Vmware Cloud Foundation 4.5.1

Cause

As the description indicates, this is due to an error within the scan workflow of the VMware Update Manager / VMware Lifecycle Manager service on the vCenter Server, due to the presence of a custom VIB installed on ESXi host(s).

Resolution

The resolution for this issue is to uninstall the offending custom VIB(s) installed on the ESXi host(s).

To identify and remove the VIB, use the following steps:

1. SSH into the vCenter Server appliance with the root user.
2. Review the vmware-vum-server.log located in /var/log/vmware/vmware-updatemgr/vum-server/
3. We should see an error entry similar to:

   • -->     <test>^M
     -->       <name>MISSING_GUNZIP_CHECKSUM_VIBS</name>^M
     -->       <expected>^M
     -->         ^M
     -->       </expected>^M
     -->       <found>^M
     -->         <value>custom-esxi_bootbank_custom-esxi_5.0.0-0.0.1</value>^M
     -->       </found>^M
     -->       <result>ERROR</result>^M
     -->     </test>^M

   • In this example, the offending VIB is: custom-esxi_bootbank_custom-esxi_5.0.0-0.0.1
4. SSH into the ESXi host(s) that we are seeing the error on.
   • Ensure that the VIB in question is safe to remove. If the custom VIB has been provided by the hardware vendor, please consult them on the consequences of removing the VIB.
5. Remove the VIB using the command:
   • esxcli software vib remove --vibname=<vibname>
6. Repeat these steps for all the affected ESXi hosts
7. Re-run the Upgrade precheck to ensure it is no longer failing at Upgrade Bundle Validation.