vSphere Lifecycle Manager Image Compliance check fails with Error "Host not compatible with the image, or Downgrades of manually added Components"

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• • The Remediate button is grayed out, preventing cluster remediation.
  • The vCenter Server Appliance (VCSA) may have recently recovered from a crash.

Environment

• VMware vCenter Server 7.x
• VMware vCenter Server 8.x
• VMware vCenter Server 9.x

Cause

vSphere Lifecycle Manager cannot perform a downgrade of components already installed on the host. This occurs if the newly configured vSphere Lifecycle Manager image includes older version VIBs than those currently installed on the ESXi host.

This issue is most often caused by a host having been installed with a vendor's customized ISO, which may contain newer versions of specific components.

This issue can also occur when the versioning for the driver on the host has changed over a long period of time such that the version in the image appears to be older than the version on the host. 
As an example:

Component	Host Version	Image Version
Cisco Ethernet native driver	CIS_bootbank_nenic_2.0.15.0-1OEM.700.1.0.15843807	2.0.10.0-5vmw.910.0.25370933

The Host Version appears to higher as it is 2.0.15 while the Image Version is 2.0.10. The "700" in the Host Version indicates that this driver was writing for ESXi 7.0 while teh "910: in the Image Version indicates that this driver was written for ESX 9.1. There has likely been a versioning shift over the large amount of time between these two ESX versions such that the 9.1 driver version uses a lower number than what was used in 7.0. Lifecycle Manager does not take this distinction into account and will flag this as a downgrade scenario.

Resolution

To resolve this issue, perform one of the following options:

Option 1: Edit the Image

1. Edit the Image and select a version of the Vendor Addon that contains higher versions of the components.
2. If the Lifecycle Manager does not contain the new version, refer to the following article to import the most recent versions: Newer versions of a vendor addon might be missing when configuring a Lifecycle Manager cluster image in vCenter Server 7.0/8.0.

3. Contact the hardware vendor for clarification on vendor add-ons and ESXi images.
4. For more information, refer to the document: Edit a vSphere Lifecycle Manager Image.

Option 2: Manually Import Bundles

1. Manually import offline bundles of the components of the same or higher version.
2. Add them to the image under the Add Components section while editing the Image.
   
3. Save the changes. Example: Adding Network driver for Intel version 1.4.3.0-1 and VMware Tools Async Release 12.5.1.

Option 3: Manually Remove VIBs Use this option if the newer version of the VIB is not necessary and an older version is desired.

1. Important: Ensure the VIB being removed is not used by Network or Storage adapters. Refer to: Determining Network/Storage firmware and driver version using ESXi Shell.
2. Place the ESXi host in Maintenance Mode.
3. Connect to the ESXi host via SSH.
4. Run the following command to list installed VIBs and identify the exact <vib_name> from the name column: 

   esxcli software vib list

5. Verify if the VIB is in use:
   • Storage drivers: 

     esxcli storage core adapter list | grep -i <vib_name>

   • Network drivers: 

     esxcli network nic list | grep -i <vib_name>

   • Driver components: 

     esxcli software profile get | grep -i <vib_name>

6. If the VIB is not in use, run the following commands to remove it: 

   esxcli software vib remove -n <vib_name>
   esxcli software component remove -n <vib_name>

7. Reboot the ESXi host.
8. Check compliance and remediate the host.

Additional Information

Upon reviewing the vSphere Lifecycle Manager (vLCM), the following message may be displayed as below:

The cluster needs remediation to finish enabling these Solutions: HBR Solution component for ESX - version 8.0.X build #### along with a note stating that remediation needs completing to finish installing the HBR VIB on the hosts

After the cluster image is made compliant and remediation is completed as outlined in the resolution steps, the HBR agent will be automatically deployed to the ESXi hosts via vSphere Lifecycle Manager (vLCM).