Enabling vSphere HA fails with error: "Applying HA VIBs on the cluster encountered a failure" due to vCenter and ESXi version mismatch.

Products

• VMware vCenter Server

Issue/Introduction

When attempting to enable or reconfigure vSphere High Availability (HA) on a cluster, the operation fails. The following symptoms are observed:

• vSphere Client Error: 

  "Cannot complete the configuration of the vSphere HA agent on the host. "Applying HA VIBs on the cluster encountered a failure".

  Failed installing HA component on the host: host-<ID> The operation cannot continue due to downgrade of the following Components: vsphere-fdm."

• Cluster Status Error: "Applying HA VIBs on the cluster encountered a failure."
  
  
  
  
  Log File: /var/run/log/lifecycle.log 

[YYYY-MM-DDTHH:MM:SS] lifecycle: 2919100: imagemanagerctl:152 ERROR [ComponentDowngradeError] (['vsphere-fdm'], 'Downgrade of Component(s) vsphere-fdm is not supported due to possible config downgrade.')
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2919100: imagemanagerctl:157 ERROR Traceback (most recent call last):
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2919100: imagemanagerctl:157 ERROR File "/usr/lib/vmware/lifecycle/bin/imagemanagerctl.py", line 670, in components
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2919100: imagemanagerctl:157 ERROR Transaction(initInstallers=False).InstallComponentsFromSources(
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2919100: imagemanagerctl:157 ERROR File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 522, in InstallComponentsFromSources
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2919100: imagemanagerctl:157 ERROR _checkComponentDowngrades(curProfile, newProfile)
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2919100: imagemanagerctl:157 ERROR File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 2264, in _checkComponentDowngrades
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2919100: imagemanagerctl:157 ERROR raise Errors.ComponentDowngradeError(
[YYYY-MM-DDTHH:MM:SS] lifecycle: 2919100: imagemanagerctl:157 ERROR vmware.esximage.Errors.ComponentDowngradeError: (['vsphere-fdm'], 'Downgrade of Component(s) vsphere-fdm is not supported due to possible config downgrade.')

Environment

• vCenter Server: Version 8.x 
• ESXi Hosts: Version 8.x 

Cause

This issue is caused by a version interoperability violation. vCenter Server expects to manage ESXi hosts of an equal or lower version.

When vSphere HA is enabled, vCenter attempts to push the Fault Domain Manager (FDM) agent VIB to the hosts. Because the ESXi hosts are on a newer update than the vCenter Server, the FDM agent provided by vCenter is viewed as an older version. The ESXi host blocks the installation because it would result in a downgrade of the vsphere-fdm component, which is not supported during the automated HA configuration process Enabling vSphere HA fails in cluster managed by Lifecycle Manager cluster image.

Resolution

To resolve this issue, align the environment versions and clear the stale agent state:

1. Upgrade vCenter Server: Upgrade the vCenter Server appliance to a version or build number that is equal to or higher than the highest ESXi build version in the cluster by following instructions from link below.
   https://knowledge.broadcom.com/external/article/316584/patchingupdating-vmware-vcenter-server-a.html
2. Re-enable HA:
   • Right-click the Cluster in the vSphere Client.
   • Navigate to Configure > vSphere Availability.
   • Click Edit, disable vSphere HA, and click OK.
   • Once the task completes, re-enable vSphere HA to force vCenter to push the correct FDM agent version.