Cannot contact the specified host error observed when adding an ESXi Host to vCenter Server

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Attempting to add a new ESXi host to vCenter fails with an error stating:

Cannot contact the specified host. The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding.

The same error may be observed if the ESXi host that is being added was previously in the vCenter's inventory, but it was removed.
Following error message observed on the vCenter UI when adding an ESXi host to vCenter:

Entries similar to the logs below will be observed in the vCenter's /var/log/vmware/vpxd/vpxd.log:

[YYYY-MM-DDTHH:MM:SS] warning vpxd[45681] [Originator@6876 sub=IO.Connection opID=lyb4w30f-126264-auto-2pff-h5:70018328-a9-01-01] Failed to connect; <io_obj p:0x00007fc43c080f08, h:84, <TCP 'vCenter_IP_ADDRESS : 46032'>, <TCP 'ESXi_IP_ADDRESS : 902'>>, e: 110(Connection timed out), duration: 129625msec
[YYYY-MM-DDTHH:MM:SS] error vpxd[43679] [Originator@6876 sub=VpxdHostConnection opID=lyb4w30f-######-auto-2pff-h5:70018328-##-##-##] Failed to connect to ESXi_IP_ADDRESS:902. Details: Connection timed out
[YYYY-MM-DDTHH:MM:SS] error vpxd[43679] [Originator@6876 sub=InvtHostCnx opID=lyb4w30f-######-auto-2pff-h5:70018328-##-##-##] Failed to determine vpxd's IP from host ESXi_IP_ADDRESS perspective: N7Vmacore15SystemExceptionE(Connection timed out)
...
[YYYY-MM-DDTHH:MM:SS] info vpxd[43679] [Originator@6876 sub=Default opID=lyb4w30f-######-auto-2pff-h5:70018328-##-##-##] [VpxLRO] -- ERROR task-134037 -- host -- Folder.addStandaloneHost: vim.fault.NoHost:
--> Result:
--> (vim.fault.NoHost) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    name = "ESXi_IP_ADDRESS or ESXi_HOSTNAME"
-->    msg = ""
--> }
--> Args:
-->
...
[YYYY-MM-DDTHH:MM:SS] info vpxd[43602] [Originator@6876 sub=vmomi.soapStub[8] opID=lyb4w30f-######-auto-2pff-h5:70018328-##_##_##-LicenseClientUnregisterHostAsync-2eb845e4] SOAP request returned HTTP failure; <SSL(<io_obj p:0x00007fc4207ba358, h:52, <TCP '127.0.0.1 : 45586'>, <TCP '127.0.0.1 : 443'>>), /ls/sdk>, method: unregisterEntity; code: 500(Internal Server Error); fault: (vim.fault.LicenseEntityNotFound) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    entityId = "host-#####"
-->    msg = "Received SOAP response fault from [<SSL(<io_obj p:0x00007fc4207ba358, h:52, <TCP '127.0.0.1 : 45586'>, <TCP '127.0.0.1 : 443'>>), /ls/sdk>]: unregisterEntity
--> "
--> }
[YYYY-MM-DDTHH:MM:SS] error vpxd[43602] [Originator@6876 sub=licenseClient opID=lyb4w30f-######-auto-2pff-h5:70018328-##_##_##-LicenseClientUnregisterHostAsync-#######] Swallowing exception 'Fault cause: vim.fault.LicenseEntityNotFound
--> ' thrown when unregister entity 'host-#####'

OR

info vpxd[06997] [Originator@6876 sub=vpxLro opID=vb-150208:urn:vmomi:ClusterComputeResource:domain-c#####:e4cb2ea1-a####938-9679-478#######.properties:01-47] [VpxLRO] -- BEGIN lro-34652189 -- ResourceModel -- vim.dp.ResourceModel.queryBatch -- 5263cb58-5ae4-5###-11b0-1bd########525######-6z-1####-d537-835a-304406######)
info vpxd[06997] [Originator@6876 sub=vpxLro opID=vb-150208:urn:vmomi:ClusterComputeResource:domain-c#####:e4c####-a9a0-4938-9679-47822cd#####.properties:01-47] [VpxLRO] -- FINISH lro-34652189
info vpxd[06927] [Originator@6876 sub=vpxLro opID=meadzlan-717705-auto-fdsa-h5:7013#####-51] [VpxLRO] -- BEGIN task-206#### -- datacenter-23701 -vim.Datacenter.batchQueryConnectInfo -- 5263cb58-5ae4-5a77-11b0-1###########(525642d5-1###-d5##-835a-30#######c839b)
warning vpxd[07004] [Originator@6876 sub=IO.Connection opID=meadzlan-717705-auto-fdsa-h5:7013#####-51-WorkQueue-40c8a97e] Failed to connect;<io_obj p:0x0000######, h:69, <TCP '###.###.##.## : 58536'>, <TCP '###.###.##.## : 443'>>, e: 111(Connection refused), duration: 0msec
warning vpxd[07004] [Originator@6876 sub=HttpConnectionPool-000001 opID=######-717705-auto-fdsa-h5:7######-51-WorkQueue-40c8a97e] Failed to get pooled connection; <cs p:00007f9######, TCP>, (null), duration: 0msec, N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)
--> [context]zKq7AVECAQAAAA8jcwENdnB4ZAAAMxxTbGlidm1hY29yZS5zbwAA/hdCAB8/QwCMmUoAKHI4AJmSOABtozgAB7I4AATsNwAXRTgAuw9RAbCOAGxpYnB0aHJlYWQuc28uMAAC3/oPbGliYy5zby42AA==[/context]
info vpxd[07004] [Originator@6876 sub=IO.Http opID=#######-######-auto-fdsa-h5:7013####-51-WorkQueue-40c8a97e] Set user agent error; state: 1, (null), N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.
)
--> [context]zKq7AVECAQAAAA8jcwENdnB4ZAAAMxxTbGlidm1hY29yZS5zbwAA/hdCAB8/QwCMmUoAKHI4AJmSOABtozgAB7I4AATsNwAXRTgAuw9RAbCOAGxpYnB0aHJlYWQuc28uMAAC3/oPbGliYy5zby42AA==[/context]
warning vpxd[06915] [Originator@6876 sub=vmomi.soapStub[4909] opID=meadzlan-717###-auto-fdsa-h5:7013#####-51-WorkQueue-40c#####] Terminating invocation; <<cs p:00007f96#######, TCP:>, /sdk>, moref: vmodl.infra.VmodlNs:infra-vmodl-ns, method: selectCapabilities
error vpxd[07004] [Originator@6876 sub=IO.Http opID=meadzlan-717705-auto-fdsa-h5:70132313-51-WorkQueue-40c8a97e] User agent failed to send request; (null), N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connection

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server
VMware ESXi

Cause

This issue can arise if there are communication failures between the vCenter Server and the ESXi host, particularly when port 902 is blocked.
This issue may also occur due to IP address conflict.

Resolution

Ensure that no firewall rules are blocking port 902 between the vCenter Server and the ESXi host, as listed in VMware Ports and Protocols
Validate the networking configuration of the hosts following KB article Multihoming on ESXi as Multihoming in vmkernel networking is not supported.
Confirm DNS resolution is working using from vCenter server to host and vice versa, using
```
nslookup <server_IP>
```

Identify if there is any duplicate IP or IP conflict of the ESXi host IP address. You can validate the duplicate IP using the below PowerCLI script:

1. Connect to an ESXi host or vCenter and enter the username and password for the administrative account:
Connect-VIServer ESXi_or_vCenter_IP -User root

2. In our case, we connect to the ESXi host managed by vCenter Server with the command:
Connect-VIServer <ESXi IP >  -User root

3. The simplest command to display a VM that has the defined IP address is:
Get-VM * |where-object{$_.Guest.IPAddress -match "<DuplicateIP>"}Where <DuplicateIP> is the IP address of the VM to be identified.

Once the duplicate IP is identified resolve the IP conflict and restart dnsmasq service on vCenter server.
- systemctl restart dnsmasq
Validate the physical networking:
- Check for bad routes on the firewall, clear the VLAN cache, fix the inconsistent MTU size, etc.
- To test connectivity to a specific port, nc can be used on both vCenter and ESXi host.
  e.g. to test TCP port 902, or 443 from the vCenter against a specific ESXi host, please do the following:
  - SSH to the vCenter.
  - Perform the test:
    # nc -zv <ESXi-host-management-IP>
    Sample of expected results:
    - Port is in "open" state: 902 (ideafarm-door) open
    - Connection was successful: 443 port [tcp/https] succeeded!