Updating ESXi Host through VCF Operations UI or SDDC Manager UI results in a pre-check warning stating "Host has third-party vibs which may be overridden"

Products

• VMware SDDC Manager
• VCF Operations

Issue/Introduction

• Unable to Patch or Upgrade ESXi Host part of Workload Domain in VCF Environment, Error: ESX Upgrade VUM Stage Scan Host task Failed
• ESXi hosts installed using a Custom Image(ISO/Offline Bundle) cannot be patched or upgraded successfully using the ESXi bundles provided in the SDDC's Bundle Management section
• Precheck will fail with Warning stating name of 3rd party vibs installed on ESXi which may lead to patch failure due to vib dependency issue :

  Host has the following third-party VIBs:[bnxtnet, bnxtroce,dellconfig, dellfac,dellfistutil,dellosname,dellupgrade, i40en,icen, igbn, irdman,ixgben-ens, ixgben,qlnativefc], which may be overridden.
  This error can be ignored if you are planning to use custom ISO to upgrade this VUM-based cluster

   

• Error Observed in SDDC Manager UI

  ESX Upgrade VUM Stage Scan Host task Failed

   

• Error Observed in vCenter by Expanding the Scan Entity Task for the Failed Host

  The upgrade has VIBs that are missing dependencies. Remove the VIBs or use Image Builder to create a custom upgrade ISO image that contains the missing dependencies, and try to upgrade again

   

• While checking logs for the vCenter Server Managing the ESXi Host we may notice similar entries, (/var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log):

  error vmware-vum-server[########] [Originator@6876 sub=VciClusterRemediateCheckTask. VciClusterRemediateCheckTask{####}] [vciClusterRemediateCheckTask 166] Found incompatible software config on host:
  ESX.EXAMPLE.COM, host Id: host-######, error: Fault cause: integrity. fault. HostUpgradeIncompatibleSoftwareConfig

Environment

VMware Cloud Foundation

Cause

The warning occurs when ESXi has 3rd Party vibs listed installed in it and while updating the ESXi using Fleet Manager/SDDC Default Vanilla Image is being used to update ESXi resulting in pre-check or compliance failure.

When ESXi Host is installed using Custom ISO there are a few OEM Bundle/vibs contained in it which are not present in Vanilla Image leading to Patch failure

Resolution

Steps to Patch ESXi Host using Custom Image:

1. Add the Vendor Addon to the Lifecycle Manager following KB: Adding Vendor ADD-ON to Lifecycle Manager
2. Create an Image on the vCenter Server managing the ESXi Host/Host's for which pre-check had warning with the Desired ESXi Build and Vendor Addon with steps mentioned below and the same can be found at Create an Image in Image Library Manually
   1. Login to vCenter Server Instance managing the Host's/Cluster
   2. Navigate to Lifecycle Manager
   3. Click on Image Library
   4. Specify the Image_Name, Desired ESXi Version, Vendor Addon Imported in Step1, any add on component(if required)
   5. Save
      
      
      
      
3. Import the Image to Fleet Manager using steps mentioned below and the same can be found at Import a vSphere Lifecycle Manager Image to a VCF Instance:
   1. Login to VCF Operations UI
   2. Navigate to Fleet Management Section
   3. Expand the Dropdown and select Lifecycle
   4. Click on the VCF_Instance section
   5. Navigate to Image Management
   6. Click on Import Image
   7. Select Import from a vCenter and select the vCenter where the Image is created in Step2
   8. Select the image created in Step2
   9. Import
      
      
      
      
4. To apply the Image on the Host part of Cluster for which the Image is created follow the steps mentioned below:
   1. Login to VCF Operations UI.
   2. Navigate to Fleet Management Section.
   3. Expand the Dropdown and select Lifecycle.
   4. Expand Dropdown on the VCF_Instance section.
   5. Click on Domain Name where the Image is created in step2.
   6. Click on Updates section.
   7. Click on Plan patching.
   8. Select ESXi as Software Component to be updated and select the Target Version to match with the version specified in the Image created in step2
   9. Click on Confirm
   10. Click on Done
   11. Click on Configure Update under Available Updates
   12. Select desired cluster where the Image is to be applied
   13. Select the Image.
   14. Click on Assign Image.
   15. Click on Next.
   16. Proceed with rest on-screen instructions.