Resolve third-party VIB preventing vSphere HA agent updates

Products

• VMware vSphere ESXi

Issue/Introduction

Provide information on how to resolve vSphere HA agent problems due to third-party VIBs.

Symptoms:

• Unable to install or update the vCenter Server vSphere High Availability (vSphere HA) agent service.
• Powering on virtual machines fails with error:
  The host is reporting errors in its attempts to provide vSphere HA support

• In the VMware vCenter Server summary, you may see this vSphere HA service error:
  vSphere HA agent for this host has an error: vSphere HA agent cannot be correctly installed or configured
   
• Manually removing the FDM agent by running the esxcli software vib remove -n vmware-fdm command may fail with an error similar to:
  [DependencyError]
  VIB Acronis_bootbank_active_drive_1.0-28 violates extensibility rule checks: [u'(line 48: col 0) Element vib failed to validate content']
  Please refer to the log file for more details.

Note: that the exact name of the third-party VIB may differ from these messages.

• In the /var/run/log/esxupdate.log file, you see entries similar to:
  bootbank_qlogic_hcli_2.2.60.7.0.0vmw.500.0.0.472560 violates extensibility rule: Installing the VIB
  fdm-installer: [24683] 2013-07-18 18:41:45: Result of esxcli software vib install -v=/tmp/vmware-root/ha-agentmgr/vpx-upgrade-installer/vmware-fdm.vib: [DependencyError]
  fdm-installer: VIB QLogic_bootbank_qlogic_hcli_2.2.60.7.0.0vmw.500.0.0.472560 violates extensibility rule checks: [u'(line 31: col 0) Element vib failed to validate content']
  fdm-installer: Please refer to the log file for more details.

Environment

VMware vSphere ESXi 6.5.x, 6.7.x, 7.0.x, 8.0.x

Cause

A third-party VIB has created a dependency for the vSphere HA agent VIB, preventing the ESXi OS from deleting the HA VIB.

Resolution

To identify the third-party VIB causing the problem:

• Review the log file for messages related to third-party VIBs:

cat /var/run/log/esxupdate.log |less

To remove the third-party VIB from the ESXi host:

1. Place the ESXi host in Maintenance Mode.
2. Disable Lockdown Mode on all hosts in the cluster.
3. Connect to the ESXi host with SSH session
4. Run this command to find the list of VIBs installed on the ESXi host:
   esxcli software vib list

Note: The exact name of the third-party VIB from the "name" column.

5. Run this command to remove the third-party VIB seen in the log messages. Replace vibname with the exact name of the third-party VIB:
   esxcli software vib remove -n vibname

To remove the vSphere HA agent VIB:

1. Follow How to run vSphere HA agent remove script in ESXi to remove vSphere HA agent from the ESXi host.
2. If the vSphere HA remove script fails, run this command to directly remove the FDM agent VIB from the ESXi host:

esxcli software vib remove -n vmware-fdm

3. Exit maintenance mode.

To reinstall the vSphere HA agent VIB:

1. Reconfigure HA on a cluster level.
2. Browse to the cluster in the vSphere Web Client object navigator.
3. Click the Manage tab and click Settings.
4. Under Services, click Edit.
5. Uncheck the Turn ON vSphere HA option.
6. Click OK.
7. Click Settings and select Turn ON vSphere HA.
8. Click OK.
9. Restart management services. For more information, see Restarting the Management agents in ESXi.
10. Disconnect and reconnect the host from the vCenter Server.

Repeat all of the steps from this article on any other affected host.

Additional Information

Error: "vSphere HA agent cannot be correctly installed or configured"

Impact/Risks:
Removing third-party VIBs may impact your ability to manage the third-party product until that VIB is reinstalled.