“Downgrades of manually added Components… are not supported” during vLCM image remediation on ESXi hosts

Products

VMware vSphere ESXi

Issue/Introduction

When using vSphere Lifecycle Manager image-based remediation to update ESXi hosts, the Run Pre-Check or image compliance check fails with an error similar to:

Host is not compatible with the image.

The vLCM UI may also report:

Downgrades of manually added Components <component-name>(<version>) in the desired Addon <vendor-addon-name> are not supported.
Add the Components of matching or higher version to the image.

Examples of impacted components may include:

Intel Virtual RAID on CPU Driver for ESXi (iavmd)
Intel RDMA auxiliary driver (irdman)
Broadcom NetXtreme-E / RoCE drivers
QLogic NetXtreme II drivers
Marvell FastLinQ / E3 / E4 network, iSCSI, FCoE, or RDMA drivers

This issue can occur when the ESXi host has newer OEM, partner async, or manually installed driver components than the versions included in the desired vLCM image or vendor addon.

Environment

VMware vSphere ESXi 8.x
VMware vCenter Server 8.x
vSphere Lifecycle Manager image-based cluster remediation

Cause

The ESXi host contains newer driver components or VIBs than the versions included in the desired vLCM image.

vSphere Lifecycle Manager image remediation does not proceed when the desired image would downgrade manually added, OEM, or partner async components. The desired image must contain matching or higher component versions, or the conflicting unused components must be removed or aligned before remediation.

This may happen when:

The host was previously installed or refreshed with an OEM custom ESXi ISO.
The host was upgraded to a newer ESXi release while retaining newer OEM/async driver components from an earlier installation.
A previous baseline, image, or manual esxcli software vib/component install operation installed newer component versions.
The desired vLCM image or vendor addon contains older versions than what is currently installed on the host.

Example:

Host installed component: MRVL-E4-CNA-Driver-Bundle 5.0.402.0-1OEM
Desired image component: MRVL-E4-CNA-Driver-Bundle 5.0.305.0-1OEM

In this condition, vLCM detects that remediation would require a component downgrade and blocks the operation.

Resolution

*IMPORTANT*

DO NOT remove or replace driver components until confirming they are not used by active hardware. Removing a driver that is bound to an active storage, network, or boot device can cause host connectivity or boot issues.

Before making changes:

Place the ESXi host into maintenance mode.
Confirm all virtual machines have been migrated or powered off.
Ensure host management access is available.
Ensure the host can be recovered if a driver removal causes unexpected impact.
Validate driver usage on each affected host individually.

Step 1: Identify the components blocking vLCM remediation

From the vLCM compliance or Full image comparison view, identify the components where the Host Version is newer than the Image Version.

On the ESXi host, collect the installed component list:
- esxcli software component list | grep -E "MRVL-E4|MRVL-E3|Broadcom-bnxt|Intel-Volume-Mgmt|Intel-irdman"
Collect the installed VIB list:
- esxcli software vib list | grep -E "bnxtnet|bnxtroce|iavmd|irdman|qfle3|qfle3f|qfle3i|qcnic|qedentv|qedrntv|qedf|qedi"

Step 2: Verify whether the drivers are in use

List the kernel modules bound to PCI hardware:
- esxcli hardware pci list | awk '/Module Name:/ {print $3}' | sort -u
Check whether the suspected modules are loaded:
- esxcli system module list | grep -E "bnxtnet|bnxtroce|iavmd|irdman|qfle3|qfle3f|qfle3i|qcnic|qedentv|qedrntv|qedf|qedi"
Check physical NIC driver usage:
- esxcli network nic list
  - Review the Driver column. If any NIC is using one of the flagged drivers, DO NOT remove that component until the hardware dependency is reviewed.
Check storage adapter driver usage:
- esxcli storage core adapter list
  - If storage adapters show drivers such as qedf, qedi, or another flagged storage driver, DO NOT remove that component until the storage dependency is reviewed.
Check NVMe controller usage if Intel VMD / VROC is involved:
- esxcli storage nvme controller list
  - If a controller or boot device is backed by iavmd, DO NOT remove or downgrade iavmd without hardware vendor validation.

Step 3: Decide whether to add matching/newer components to vLCM or remove unused stale components

Use one of the following supported approaches:

Option A: Add matching or newer components to the vLCM image.

*Use this option when the driver is required or is actively bound to hardware.

Download the matching or newer supported offline bundle/component for the ESXi release.
- "Manually added component <VIB Name> is an unsupported downgrade from version <version>
Import the offline bundle into the vLCM depot.
Edit the cluster image.
Add the matching or newer component under Components.
Validate the image.
Run vLCM compliance/pre-check again.
Remediate the host or cluster.

This aligns the desired image with the currently installed host component version and avoids a downgrade.

Option B: Remove unused stale components from the ESXi host.

*Use this option only when the driver/component is confirmed not to be used by active hardware.

Place the host into maintenance mode.
Enable SSH on the ESXi host.
Validate that the flagged components are not bound to physical NICs, storage adapters, NVMe controllers, or boot devices.
Remove only the unused components.
- Example commands:
  - esxcli software component remove -n MRVL-E4-CNA-Driver-Bundle
  - esxcli software component remove -n MRVL-E3-Ethernet-iSCSI-FCoE
  - esxcli software component remove -n Intel-irdman
  - esxcli software component remove -n Broadcom-bnxt-Net-RoCE
Reboot the host.
- reboot
After the host returns, rerun the vLCM Run Pre-Check.
Continue with normal vLCM image remediation if the pre-check succeeds.

Step 4 (If needed): Handling components marked required by the vendor addon

Some components may not be removable because the vendor addon marks them as required. For example, removing Intel-Volume-Mgmt-Device may fail with:

[ProfileValidationError]
Profile <profile-name> is missing component(s) Intel-Volume-Mgmt-Device which are necessary and removing them is prohibited.
Please ensure that these components are fully installed or overridden without partial VIB replacement or removal.

*In this condition, do not continue trying to remove the required component as a normal component removal.

Instead, align the component with a supported version by either:

Adding a matching or higher supported version of the component to the vLCM image, or
Installing the vendor/addon-compatible VIB version if directed by the vendor guidance and after validating it is appropriate for the ESXi version and hardware.
- Example:
  - esxcli software vib install -v "/vmfs/volumes/<datastore>/<path>/INT_bootbank_iavmd_<version>.vib"
After installing or aligning the required VIB:
- reboot
Rerun the vLCM Run Pre-Check.

Additional Information

Component to driver mapping examples:

Component	Common driver/VIB names	Description
`Broadcom-bnxt-Net-RoCE`	`bnxtnet`, `bnxtroce`	Broadcom NetXtreme-E network/RoCE drivers
`Intel-Volume-Mgmt-Device`	`iavmd`	Intel VMD / Intel VROC driver
`Intel-irdman`	`irdman`	Intel X722/E810 RDMA auxiliary driver
`MRVL-E3-Ethernet-iSCSI-FCoE`	`qfle3`, `qfle3f`, `qfle3i`, `qcnic`	QLogic/Marvell NetXtreme II E3 drivers
`MRVL-E4-CNA-Driver-Bundle`	`qedentv`, `qedrntv`, `qedf`, `qedi`	Marvell FastLinQ E4 network/RDMA/FCoE/iSCSI drivers