Enabling vSphere HA fails in cluster managed by Lifecycle Manager cluster image

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
Enabling vSphere HA fails in a cluster managed by vCenter Lifecycle Manager cluster Image with the below error:
"configuration of the vSphere HA agent on the host. "Applying HA VIBs on the cluster encountered a failure"

In the latest /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server-<sequence>.log, you may see the below error in the VCSA.

2021-05-19T11:16:02.031Z info vmware-vum-server[15911] [Originator@6876 sub=ClusterApplyHATask] Task:com.vmware.vcIntegrity.lifecycle.ClusterApplyHATask ID:52543c46-ee7a-3920-4529-dda799cf9d1c. Spawned HA Apply task(5258a290-a6c5-b092-61b9-074dcb72b2e9) on host(host-54)
2021-05-19T11:16:14.026Z error vmware-vum-server[15911] [Originator@6876 sub=ClusterApplyHATask] Task:com.vmware.vcIntegrity.lifecycle.ClusterApplyHATask ID:52543c46-ee7a-3920-4529-dda799cf9d1c. Host HA apply task(5258a290-a6c5-b092-61b9-074dcb72b2e9) failed on host(host-54) 

On the impacted ESXi Host lifecycle.log, you may see the below:
Note: lifecycle.log is located at /var/run/log/lifecycle.log on the impacted ESXi Host.

2021-05-19T11:37:10Z lifecycle: 2919100: imagemanagerctl:152 ERROR [ComponentDowngradeError] (['vsphere-fdm'], 'Downgrade of Component(s) vsphere-fdm is not supported due to possible config downgrade.')
2021-05-19T11:37:10Z lifecycle: 2919100: imagemanagerctl:157 ERROR Traceback (most recent call last):
2021-05-19T11:37:10Z lifecycle: 2919100: imagemanagerctl:157 ERROR File "/usr/lib/vmware/lifecycle/bin/imagemanagerctl.py", line 670, in components
2021-05-19T11:37:10Z lifecycle: 2919100: imagemanagerctl:157 ERROR Transaction(initInstallers=False).InstallComponentsFromSources(
2021-05-19T11:37:10Z lifecycle: 2919100: imagemanagerctl:157 ERROR File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 522, in InstallComponentsFromSources
2021-05-19T11:37:10Z lifecycle: 2919100: imagemanagerctl:157 ERROR _checkComponentDowngrades(curProfile, newProfile)
2021-05-19T11:37:10Z lifecycle: 2919100: imagemanagerctl:157 ERROR File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 2264, in _checkComponentDowngrades
2021-05-19T11:37:10Z lifecycle: 2919100: imagemanagerctl:157 ERROR raise Errors.ComponentDowngradeError(
2021-05-19T11:37:10Z lifecycle: 2919100: imagemanagerctl:157 ERROR vmware.esximage.Errors.ComponentDowngradeError: (['vsphere-fdm'], 'Downgrade of Component(s) vsphere-fdm is not supported due to possible config downgrade.')

Environment

VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 7.0.1

Cause

This issue happens in one of two scenarios:

1. vCenter is restored (from backup) that was on lower vCenter version than the current one .
2. ESXi Host(s) is moved from higher vCenter version to a lower one.

The reason of the failure is image code on ESXi lack the ability to handle FDM downgrade when config store is utilized by the VIB internally.

Resolution

This issue is fixed in ESXi 7.0U2c. For more information, see VMware ESXi 7.0 Update 2c Release Notes.

Workaround:
To workaround this issue, follow the below steps:

1. Connect to the impacted ESXi Host(s) through SSH.
2. Run the below command:

esxcli software vib remove -n vmware-fdm

3. Enable vSphere HA again.