Error 'Cannot Contact the Specified Host' When Adding an ESXi Host to vCenter Server

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Attempting to add a new ESXi host to vCenter fails with an error stating:

Cannot contact the specified host. The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding.

/var/log/vmware/vpxd/vpxd.log you will see entries similar to:

[YYYY-MM-DDTHH:MM:SS] warning vpxd[45681] [Originator@6876 sub=IO.Connection opID=lyb4w30f-126264-auto-2pff-h5:70018328-a9-01-01] Failed to connect; <io_obj p:0x00007fc43c080f08, h:84, <TCP 'vCenter_IP_ADDRESS : 46032'>, <TCP 'ESXi_IP_ADDRESS : 902'>>, e: 110(Connection timed out), duration: 129625msec
[YYYY-MM-DDTHH:MM:SS] error vpxd[43679] [Originator@6876 sub=VpxdHostConnection opID=lyb4w30f-######-auto-2pff-h5:70018328-##-##-##] Failed to connect to ESXi_IP_ADDRESS:902. Details: Connection timed out
[YYYY-MM-DDTHH:MM:SS] error vpxd[43679] [Originator@6876 sub=InvtHostCnx opID=lyb4w30f-######-auto-2pff-h5:70018328-##-##-##] Failed to determine vpxd's IP from host ESXi_IP_ADDRESS perspective: N7Vmacore15SystemExceptionE(Connection timed out)
...
[YYYY-MM-DDTHH:MM:SS] info vpxd[43679] [Originator@6876 sub=Default opID=lyb4w30f-######-auto-2pff-h5:70018328-##-##-##] [VpxLRO] -- ERROR task-134037 -- host -- Folder.addStandaloneHost: vim.fault.NoHost:
--> Result:
--> (vim.fault.NoHost) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    name = "ESXi_IP_ADDRESS or ESXi_HOSTNAME"
-->    msg = ""
--> }
--> Args:
-->
...
[YYYY-MM-DDTHH:MM:SS] info vpxd[43602] [Originator@6876 sub=vmomi.soapStub[8] opID=lyb4w30f-######-auto-2pff-h5:70018328-##_##_##-LicenseClientUnregisterHostAsync-2eb845e4] SOAP request returned HTTP failure; <SSL(<io_obj p:0x00007fc4207ba358, h:52, <TCP '127.0.0.1 : 45586'>, <TCP '127.0.0.1 : 443'>>), /ls/sdk>, method: unregisterEntity; code: 500(Internal Server Error); fault: (vim.fault.LicenseEntityNotFound) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    entityId = "host-#####"
-->    msg = "Received SOAP response fault from [<SSL(<io_obj p:0x00007fc4207ba358, h:52, <TCP '127.0.0.1 : 45586'>, <TCP '127.0.0.1 : 443'>>), /ls/sdk>]: unregisterEntity
--> "
--> }
[YYYY-MM-DDTHH:MM:SS] error vpxd[43602] [Originator@6876 sub=licenseClient opID=lyb4w30f-######-auto-2pff-h5:70018328-##_##_##-LicenseClientUnregisterHostAsync-#######] Swallowing exception 'Fault cause: vim.fault.LicenseEntityNotFound
--> ' thrown when unregister entity 'host-#####'

OR

info vpxd[06997] [Originator@6876 sub=vpxLro opID=vb-150208:urn:vmomi:ClusterComputeResource:domain-c#####:e4cb2ea1-a####938-9679-478#######.properties:01-47] [VpxLRO] -- BEGIN lro-34652189 -- ResourceModel -- vim.dp.ResourceModel.queryBatch -- 5263cb58-5ae4-5###-11b0-1bd########525######-6z-1####-d537-835a-304406######)
info vpxd[06997] [Originator@6876 sub=vpxLro opID=vb-150208:urn:vmomi:ClusterComputeResource:domain-c#####:e4c####-a9a0-4938-9679-47822cd#####.properties:01-47] [VpxLRO] -- FINISH lro-34652189
info vpxd[06927] [Originator@6876 sub=vpxLro opID=meadzlan-717705-auto-fdsa-h5:7013#####-51] [VpxLRO] -- BEGIN task-206#### -- datacenter-23701 -vim.Datacenter.batchQueryConnectInfo -- 5263cb58-5ae4-5a77-11b0-1###########(525642d5-1###-d5##-835a-30#######c839b)
warning vpxd[07004] [Originator@6876 sub=IO.Connection opID=meadzlan-717705-auto-fdsa-h5:7013#####-51-WorkQueue-40c8a97e] Failed to connect;<io_obj p:0x0000######, h:69, <TCP '###.###.##.## : 58536'>, <TCP '###.###.##.## : 443'>>, e: 111(Connection refused), duration: 0msec
warning vpxd[07004] [Originator@6876 sub=HttpConnectionPool-000001 opID=######-717705-auto-fdsa-h5:7######-51-WorkQueue-40c8a97e] Failed to get pooled connection; <cs p:00007f9######, TCP>, (null), duration: 0msec, N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)
--> [context]zKq7AVECAQAAAA8jcwENdnB4ZAAAMxxTbGlidm1hY29yZS5zbwAA/hdCAB8/QwCMmUoAKHI4AJmSOABtozgAB7I4AATsNwAXRTgAuw9RAbCOAGxpYnB0aHJlYWQuc28uMAAC3/oPbGliYy5zby42AA==[/context]
info vpxd[07004] [Originator@6876 sub=IO.Http opID=#######-######-auto-fdsa-h5:7013####-51-WorkQueue-40c8a97e] Set user agent error; state: 1, (null), N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.
)
--> [context]zKq7AVECAQAAAA8jcwENdnB4ZAAAMxxTbGlidm1hY29yZS5zbwAA/hdCAB8/QwCMmUoAKHI4AJmSOABtozgAB7I4AATsNwAXRTgAuw9RAbCOAGxpYnB0aHJlYWQuc28uMAAC3/oPbGliYy5zby42AA==[/context]
warning vpxd[06915] [Originator@6876 sub=vmomi.soapStub[4909] opID=meadzlan-717###-auto-fdsa-h5:7013#####-51-WorkQueue-40c#####] Terminating invocation; <<cs p:00007f96#######, TCP:>, /sdk>, moref: vmodl.infra.VmodlNs:infra-vmodl-ns, method: selectCapabilities
error vpxd[07004] [Originator@6876 sub=IO.Http opID=meadzlan-717705-auto-fdsa-h5:70132313-51-WorkQueue-40c8a97e] User agent failed to send request; (null), N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connection

Environment

VMware vCenter Server 8.x

Cause

This issue can arise if there are communication failures between the vCenter Server and the ESXi host, particularly when port 902 is blocked or it can occur due to IP address conflict.

Resolution

Ensure that no firewall rules are blocking port 902 between the vCenter Server and the ESXi host, as listed in VMware Ports and Protocols
Validate the networking configuration of the hosts following this article Multihoming on ESXi as Multihoming in VMkernel networking is not supported.
Confirm DNS resolution is working using from vCenter server to host and vice versa, using
```
nslookup <server_IP>
```

Identify if there is any duplicate IP or IP conflict of the ESXi host IP address. You can validate the duplicate IP using the below PowerCLI script:

1. Connect to an ESXi host or vCenter and enter the username and password for the administrative account:
Connect-VIServer ESXi_or_vCenter_IP -User root

2. In our case, we connect to the ESXi host managed by vCenter Server with the command:
Connect-VIServer <ESXi IP >  -User root

3. The simplest command to display a VM that has the defined IP address is:
Get-VM * |where-object{$_.Guest.IPAddress -match "<DuplicateIP>"}Where <DuplicateIP> is the IP address of the VM to be identified.

Once the duplicate IP is identified resolve the IP conflict and restart dnsmasq service on vCenter server.
- systemctl restart dnsmasq
Validate the physical networking:
- Check for bad routes on the firewall, clear the VLAN cache, etc.