HA agent configuration fails with error: Applying HA VIBs on the cluster encountered a failure

Products

VMware vSphere ESXi VMware vCenter Server

Issue/Introduction

When configuring vSphere HA on a cluster, the configuration fails for ESXi hosts with below error.

Manually installing the FDM VIB on the ESXi host, or disconnecting and reconnecting the host followed by enabling vSphere HA, fails with the same error. Port connectivity over 902 and 443 between vCenter Server (VCSA) and ESXi shows as successful.

The /var/log/vmware/vpxd/vpxd.log on the vCenter Server displays the following HTTP 503 and 400 errors:

YYYY-MM-DDTHH:MM:SS40Z info vpxd[05563] [Originator@6876 sub=Vmomi opID=FdmMonitor-domain-cxx-xxxxxxxx] Creating SOAP stub adapter for /fdm on <ESXI-IP>:443
YYYY-MM-DDTHH:MM:SS40Z warning vpxd[05563] [Originator@6876 sub=Vmomi opID=FdmMonitor-domain-cxx-xxxxxxxx] [FdmClientAdapter] Login request throttled on host <ESXI-IP>. clientAdapterVersion: <csi.version.v9_0_0_0, official, 9.0.0.0>
YYYY-MM-DDTHH:MM:SS40Z error vpxd[05563] [Originator@6876 sub=Vmomi opID=FdmMonitor-domain-cxx-xxxxxxxx] Cannot contact the specified host <ESXI-IP>
YYYY-MM-DDTHH:MM:SS40Z error vpxd[05563] [Originator@6876 sub=HostUpgrader opID=FdmMonitor-domain-cxx-xxxxxxxx] Failed to get fdm aboutInfo from host-xx: N5Vmomi5Fault17HostCommunication9ExceptionE(Fault cause: vmodl.fault.HostCommunication
--> )
--> [context]zKq7AVECAQAAAA7SfAEXdnB4ZAAAyzdIbGlidm1hY29yZS5zbwAArDA3AA9VOIHzriwBdnB4ZACByJY8AYGFEl8BgQwUXwGBi65eAYEP3F4BAkfVGmxpYnZtb21pLnNvAAO//QRsaWJjc2ktdHlwZXMuc28Agbg3UQGBlf7JAYGFBMoBgQHRxAGB4HurAoGffKsCAP9gLQAE/CwAD1UtAOsqRgTEkAhsaWJjLnNvLjYABGyREA==[/context] - assuming no upgrade needed
YYYY-MM-DDTHH:MM:SS41Z info vpxd[05563] [Originator@6876 sub=HostUpgrader opID=FdmMonitor-domain-cxx-xxxxxxxx] Checking FDM info with legacy stub version - host-xx
YYYY-MM-DDTHH:MM:SS41Z info vpxd[05563] [Originator@6876 sub=HostUpgrader opID=FdmMonitor-domain-cxx-xxxxxxxx] Renewing Fdm client adapter version; toVersion:csi.version.version1
YYYY-MM-DDTHH:MM:SS41Z info vpxd[05563] [Originator@6876 sub=Vmomi opID=FdmMonitor-domain-cxx-xxxxxxxx] [FdmClientAdapter] Renewed the adapter; oldVersion: csi.version.v9_0_0_0, newVersion: csi.version.version1
YYYY-MM-DDTHH:MM:SS41Z info vpxd[05563] [Originator@6876 sub=Vmomi opID=FdmMonitor-domain-cxx-xxxxxxxx] Creating SOAP stub adapter for /fdm on <ESXI-IP>:443
YYYY-MM-DDTHH:MM:SS43Z info vpxd[05563] [Originator@6876 sub=vmomi.soapStub[1162] opID=FdmMonitor-domain-cxx-xxxxxxxx] SOAP request returned HTTP failure; <<io_obj t:N7Vmacore6System21LocalSocketObjectAsioE, h:52, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-xx/fdm>, method: login; code: 400(Bad Request); fault: (null)
YYYY-MM-DDTHH:MM:SS43Z error vpxd[05563] [Originator@6876 sub=Vmomi opID=FdmMonitor-domain-cxx-xxxxxxxx] Failed to login on host; <</hgw/host-xx>, /fdm>, N7Vmacore4Http13HttpExceptionE(HTTP error response: Bad Request)

Environment

VMware vCenter Server 9.0.x

VMware vCenter Server 8.0.x

VMware vSphere ESXi 8.0.x

Cause

The Envoy Host Gateway (hgw-pipe) on the ESXi host actively rejects the vCenter login request. This occurs because the API payloads are incompatible or because a manual Fault Domain Manager (FDM) VIB installation was performed without restarting the host's management agents, leaving the /fdm endpoint in a stale or unavailable state.

Resolution

In the vSphere Client, turn off vSphere HA for the cluster temporarily and place the affected ESXi host into Maintenance Mode.
Open an SSH session to the affected ESXi host and remove the incompatible FDM agent:
Note: This is not applicable from ESX 9.0 onwards.

Bash
```
esxcli software vib list | grep fdm

esxcli software vib remove -n vmware-fdm
```
Restart the management agents to clear out the stale Envoy proxy configurations and free up the /fdm endpoint:
```
services.sh restart
```
Return to the vSphere Client. If HA was disabled at the cluster level, re-enable it. If you placed the host in Maintenance Mode, right-click the specific ESXi host and select Reconfigure for vSphere HA. vCenter will now automatically push the correct VIB.
Once HA shows as "Connected" or "Running," exit maintenance mode.

Note: Alternatively, rebooting the affected ESXi host will also restart the management agents and clear the stale configurations, allowing vSphere HA to be installed successfully.

Additional Information

vSphere High Availability (HA) issues

Removing vSphere HA FDM vib from ESXi