Adding a VMware ESXi/ESX host to VMware vCenter Server fails

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Cannot connect or reconnect the ESX\ESXi host to vCenter Server.
• Adding an ESX\ESXi host to vCenter Server fails with the error:
  
  Cannot contact the specified host hostname\IP. The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding.
  
  
• After upgrading vCenter Server, a host does not reconnect to vCenter Server.
• see the error similar to:
  
  
  • Unable to access the specified host. It either does not exist, the server software is not responding, or there is a network problem.
  • Unable to add host to vCenter.

• In the /var/run/log/vpxa.log file, entries look similar to:
  
  'App' 7338928 error] [VpxaHalCnxHostagent] Failed to discover namespace: Success
  'App' 7338928 warning] [VpxaHalCnxHostagent] Could not resolve namespace for authenticating to host agent
  'App' 4283312 verbose] [SchedulePolling] Last stats polling used [0] ms
  'App' 3076461728 verbose] [VpxaHalCnxHostagent] Creating temporary connect spec: localhost:443
  'App' 3076461728 error] [VpxaHalCnxHostagent] Failed to discover namespace: Success
  'App' 3076461728 warning] [VpxaHalCnxHostagent] Could not resolve namespace for authenticating to host agent
  'App' 5815216 verbose] [SchedulePolling] Last stats polling used [0] ms
  'App' 4017072 verbose] [VpxaHalCnxHostagent] Creating temporary connect spec: localhost:443

• In the /var/run/log/hostd.log file, entries look similar to:
  
  
  • F66C36D0 panic 'App'] error: Invalid argument : No IP addresses found for addr='localhost', port=12001 : (0) 'Unknown error'
    F66C36D0 panic 'App'] backtrace:
    [00] eip 0xf7d21965
    [01] eip 0xf7c0e5c0
    
  • <YYYY-MM-DD>T<time>Z [FF8B5D20 info 'Solo'] webServer/port: 8309
    *** In-memory logs end ***
    error: N7Vmacore24InvalidArgumentExceptionE(Invalid argument : No IP addresses found for addr='localhost', port=8309 : (0) 'Unknown error')
    backtrace:
    backtrace[00] rip 16cff6e3 Vmacore::System::Stacktrace::CaptureWork(unsigned int)
    backtrace[01] rip 16b24268 Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
    backtrace[02] rip 16ac2c95 Vmacore::Throwable::Throwable(std::string const&)
    backtrace[03] rip 16cc2b61 Vmacore::System::TCPServerSocketPosix::BindAndListen(std::string const&, int)
    backtrace[04] rip 050c4558 hostd [0x50c4558]
    backtrace[05] rip 050c87f0 hostd [0x50c87f0]
    backtrace[06] rip 050cbde6 hostd [0x50cbde6]
    backtrace[07] rip 16ae141b Vmacore::Service::AppImpl::StartPlugins()
    backtrace[08] rip 16adfd3f Vmacore::Service::InitApp(Vmacore::Service::Config*)
    backtrace[09] rip 05d4fe33 hostd [0x5d4fe33]
    backtrace[10] rip 05d45a28 hostd [0x5d45a28]
    backtrace[11] rip 05d53e52 hostd [0x5d53e52]
    backtrace[12] rip 1c886efc /lib/libc.so.6(__libc_start_main+0xdc) [0x1c886efc]
    backtrace[13] rip 050c4141 hostd [0x50c4141]</time>

Note: The preceding log excerpts are only examples. 

Environment

• VMware vSphere ESXi 6.X
• VMware vSphere ESXi 7.X
• VMware vSphere ESXi 8.X

Cause

The issue can occur due to asymmetric network routing configuration on the ESXi Host

Misconfiguration can result in unidirectional communication, preventing successful TCP handshakes between vCenter and the ESXi hosts on port 902

Resolution

Confirm port 902 is open between vCenter and ESXi by running the following commands to test: 

1. From vCenter to ESXi host:
   • • nc -zv <host_ip> 902
   • Successful output of the command will be as follows:
     • root@vcsa02 [ ~ ]# nc -zv host_fqdn/ip 902 
host_fqdn [Host_ip] 902 (ideafarm-door) open -> Output of the command

• •  

1. • (OR )run the following command:
     • curl -v telnet://hostd_fqdn/IP:902
       • root@vcsa02 [ ~ ]# curl -v telnet://host_fqdn/ip:902
         * Trying host_ip:902...
         * Connected to host_fqdn (host_ip) port 902 (#0)
         220 VMware Authentication Daemon Version 1.10: SSL Required, ServerDaemonProtocol:SOAP, MKSDisplayProtocol:VNC , VMXARGS supported, NFCSSL supported/t, SHA256 supported
2. From ESXi to the vCenter:
   • nc -zu <vc_ip/fqdn> 902
   • Successful output of the following:
     • [root@esxi07:~] nc -zu vc_fqdn/ip 902
       Connection to vc_fqdn/ip 902 port [udp/*] succeeded!
3. Ensure that the /etc/hosts file is using 127.0.0.1 pointing to localhost.localdomain localhost.
   • For more information, see:
     • Editing files on an ESX host using vi or nano
     • Identifying issues with and setting up name resolution on ESX Server
4. Ensure that the host field within the /etc/nsswitch.conf file is set to hosts:files. Reboot the host for the changes to take effect.
   Note: If using DNS for authentication, ensure that the hosts field is set to hosts:dns files.
   • • Restart the Management agents on the  ESXi/ESX Server and add the host to vCenter Server. Restarting the Management agents on an ESX Server.
     • Reboot the ESXI host if in case the issue still persists and try to connect again.
5. Validate VMkernel Network Traffic
   
   • Ensure that management traffic is flowing through the VMkernel adaptor with the Management Service enabled (typically vmk0).
   • If incoming packets are received on vmk0 but outgoing packets are sent via vmk2 or vmk3 (commonly used for services such as vSphere Replication, vMotion, or other applications), it may result in communication failures and host disconnections. (Packet capture on ESXi using the pktcap-uw tool)
   • Remove any duplicate or unnecessary VMkernel adaptors configured on the ESXi host to avoid conflicts.