Configuring vSphere HA fails at 99% with errors - Operation timed out and vSphere HA agent cannot be installed or configured

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

This article provides information on why vSphere HA configuration fails to install or configure vSphere HA agent on the ESXi after rolling back the ESXi from 6.7 to the previous version

Symptoms:

• vSphere HA configuration fails at 99% as Operation timed out
• You see vSphere HA agent cannot be installed or configured error

The ESXi will have "vmware-fdm" of a different version and will fail to uninstall, preventing the installation of the latest FDM vib from the vCenter to manage the ESXi. 

vmware-fdm vib removal will fail with error message as below:

[root@esxhost:~] esxcli software vib remove -n vmware-fdm
 [DependencyError]
 VIB VMware_locker_tools-light_10.3.2.9925305-10176879 requires esx-version >= 6.6.0, but the requirement cannot be satisfied within the ImageProfile.
 Please refer to the log file for more details.

Environment

VMware vCenter Server Appliance 6.7.x
VMware vSphere ESXi 6.7
VMware vCenter Server 6.7.x
VMware vSphere ESXi 6.0
VMware vSphere ESXi 6.5

Cause

After the ESXi rollback from version 6.7, the new tools-light VIB does not revert to the earlier version. As a result, the VIB becomes incompatible with the rolled back ESXi host causing these issues. Ref (https://techdocs.broadcom.com/content/dam/broadcom/techdocs/us/en/pdf/vmware/vsphere/vsphere/vSphere-6-7-Release-Notes/vsphere-esxi-vcenter-server-67-release-notes.pdf)

Because of the above behavior the "vmware-fdm" vib on the ESXi fails removal, preventing the installation of the latest FDM vib from vCenter causing the vSphere HA to fail with error "vSphere HA agent cannot be installed or configured"

Resolution

To resolve the issue

1. Find the current version of the ESXi using the command vmware -vl
2. Download the VMware tools vib from https://support.broadcom.com/ for the above build
3. Upload the vib or the zip to the ESXi and install the "VMware-Tools" for the above build using below command
   
   [root@esxhost:/vmfs/volumes/55959755-b9c93df6-bb0d-3c4a9276bb7c] esxcli software vib install -v "/vmfs/volumes/55959755-b9c93df6-bb0d-3c4a9276bb7c/VMware_locker_tools-light_6.5.0-10.3.5.10430147.vib" Installation Result    Message: Operation finished successfully.    Reboot Required: false    VIBs Installed: VMware_locker_tools-light_6.5.0-10.3.5.10430147    VIBs Removed: VMware_locker_tools-light_10.3.2.9925305-10176879    VIBs Skipped:
    
4. Now remove the vmware-fdm vib using the following command esxcli software vib remove -n vmware-fdm, the vmware-fdm should now uninstall successfully
   
   Note: Perform the above step manually on all the affected ESXi hosts, however, this is not applicable from ESX 9.0 onwards.
    
5. Configure vSphere HA 

Additional Information

https://techdocs.broadcom.com/content/dam/broadcom/techdocs/us/en/pdf/vmware/vsphere/vsphere/vSphere-6-7-Release-Notes/vsphere-esxi-vcenter-server-67-release-notes.pdf