ESXi upgrade fails with "Host in Incompatible Compliance State"

Products

• VMware Cloud Foundation

Issue/Introduction

• When initiating the ESXi upgrade from VCF Operations, the "Proceed with scheduling the upgrade once precheck execution is complete" task fails. 
• In vCenter Server, the task of "Remediate Cluster" fails with the following error:
  
  A general system error occurred: "Cluster '<cluster name>' remediation failed because hosts '<host fqdn>' is in 'incompatible' compliance state."
  
  
• Checking the detailed compliance status under vCenter --> select [Cluster name] --> Updates --> Image --> Image Compliance, below error is reported:
  
  Downgrades of Addon Components <component name> <version> in the desired ESXi version are not supported.

Environment

VMware Cloud Foundation 9.1.0
VMware Cloud Foundation 9.0.x

Cause

VCF Operations orchestrates ESXi upgrades by leveraging the native vCenter vLCM workflow. This failure occurs because the vLCM image compliance check detects an implicit downgrade of an asynchronous component (typically third-party vendor add-ons or hardware drivers) currently installed on the physical ESXi host compared to what is specified in the target upgrade image.

As vLCM explicitly restricts the automated downgrade of these components during an image-based remediation, the upgrade workflow is safely blocked.

Resolution

To resolve the issue, update the target vLCM image to correctly include or match the required vendor add-on version, synchronize it with VCF Operations, and retry the upgrade.

Step1: Reconfigure the cluster image in vCenter Server:

1. Log in to vSphere Client.
2. Go to Lifecycle Manager, click "Create Image".
3. Enter Image Name, select ESXi base image version, select Vendor Addon and Firmware and Drivers Addon(if any).
4. Click "Validate" and "Save" if the image is validated.

Step 2: Import the updated image to VCF Operations:

1. Log in to VCF Operations.
2. Go to Build -->Lifecycle -->VCF Instances -->vcfinstance -->Image Management, click "Import Image".
3. On "Import Image" page, select "Import from a vCenter", select the newly created image in step1 and click "Import".

Step 3: Re-initiate the upgrade from VCF Operations:

1. Return to the VCF Operations upgrade wizard.
2. Configure the ESXi upgrade plan and assign the imported image in step 2 to the cluster or the ESXi host that needs to be upgraded.
3. Re-run the upgrade prechecks, proceed with scheduling and upgrade once precheck completes.