Adding a VMware ESXi/ESX host to VMware vCenter Server fails

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Cannot connect or reconnect the ESX\ESXi host to vCenter Server.
• Adding an ESX\ESXi host to vCenter Server fails with the error:
  
  Cannot contact the specified host hostname\IP. The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding.
  
  
• After upgrading vCenter Server, a host does not reconnect to vCenter Server.
• see the error similar to:
  
  
  • Unable to access the specified host. It either does not exist, the server software is not responding, or there is a network problem.
  • Unable to add host to vCenter.
    
• In the /var/run/log/vpxa.log file, entries look similar to:
  
  'App' 7338928 error] [VpxaHalCnxHostagent] Failed to discover namespace: Success
  'App' 7338928 warning] [VpxaHalCnxHostagent] Could not resolve namespace for authenticating to host agent
  'App' 4283312 verbose] [SchedulePolling] Last stats polling used [0] ms
  'App' 3076461728 verbose] [VpxaHalCnxHostagent] Creating temporary connect spec: localhost:443
  'App' 3076461728 error] [VpxaHalCnxHostagent] Failed to discover namespace: Success
  'App' 3076461728 warning] [VpxaHalCnxHostagent] Could not resolve namespace for authenticating to host agent
  'App' 5815216 verbose] [SchedulePolling] Last stats polling used [0] ms
  'App' 4017072 verbose] [VpxaHalCnxHostagent] Creating temporary connect spec: localhost:443
  
• In the /var/run/log/hostd.log file, entries look similar to:
  
  
  • F66C36D0 panic 'App'] error: Invalid argument : No IP addresses found for addr='localhost', port=12001 : (0) 'Unknown error'
    F66C36D0 panic 'App'] backtrace:
    [00] eip 0xf7d21965
    [01] eip 0xf7c0e5c0
    
  • <YYYY-MM-DD>T<time>Z [FF8B5D20 info 'Solo'] webServer/port: 8309
    *** In-memory logs end ***
    error: N7Vmacore24InvalidArgumentExceptionE(Invalid argument : No IP addresses found for addr='localhost', port=8309 : (0) 'Unknown error')
    backtrace:
    backtrace[00] rip 16cff6e3 Vmacore::System::Stacktrace::CaptureWork(unsigned int)
    backtrace[01] rip 16b24268 Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
    backtrace[02] rip 16ac2c95 Vmacore::Throwable::Throwable(std::string const&)
    backtrace[03] rip 16cc2b61 Vmacore::System::TCPServerSocketPosix::BindAndListen(std::string const&, int)
    backtrace[04] rip 050c4558 hostd [0x50c4558]
    backtrace[05] rip 050c87f0 hostd [0x50c87f0]
    backtrace[06] rip 050cbde6 hostd [0x50cbde6]
    backtrace[07] rip 16ae141b Vmacore::Service::AppImpl::StartPlugins()
    backtrace[08] rip 16adfd3f Vmacore::Service::InitApp(Vmacore::Service::Config*)
    backtrace[09] rip 05d4fe33 hostd [0x5d4fe33]
    backtrace[10] rip 05d45a28 hostd [0x5d45a28]
    backtrace[11] rip 05d53e52 hostd [0x5d53e52]
    backtrace[12] rip 1c886efc /lib/libc.so.6(__libc_start_main+0xdc) [0x1c886efc]
    backtrace[13] rip 050c4141 hostd [0x50c4141]</time>

Note: The preceding log excerpts are only examples. 

Environment

• VMware vSphere ESXi 6.X
• VMware vSphere ESXi 7.X
• VMware vSphere ESXi 8.X

Cause

https://knowledge.broadcom.com/external/article?legacyId=1020302

Resolution

 

Confirm port 902 is open between vCenter and ESXi by running the following commands to test: 

• From vCenter to ESXi host:
  • • nc -zv <host_ip> 902

         Successful output of the command will be as follows:

• • • root@vcsa02 [ ~ ]# nc -zv host_fqdn/ip 902 
      host_fqdn [Host_ip] 902 (ideafarm-door) open -> Output of the command

• • (OR )run the following command:
    • curl -v telnet://hostd_fqdn/IP:902
      • root@vcsa02 [ ~ ]# curl -v telnet://host_fqdn/ip:902
        * Trying host_ip:902...
        * Connected to host_fqdn (host_ip) port 902 (#0)
        220 VMware Authentication Daemon Version 1.10: SSL Required, ServerDaemonProtocol:SOAP, MKSDisplayProtocol:VNC , VMXARGS supported, NFCSSL supported/t, SHA256 supported

 

• From ESXi to the vCenter:
  • • nc -zu <vc_ip/fqdn> 902
    • Successful output of the following:
      • [root@esxi07:~] nc -zu vc_fqdn/ip 902
        Connection to vc_fqdn/ip 902 port [udp/*] succeeded!

 

• Ensure that the /etc/hosts file is using 127.0.0.1 pointing to localhost.localdomain localhost.
  
  For more information, see:
  
  
  • Editing files on an ESX host using vi or nano (1020302)
  • Identifying issues with and setting up name resolution on ESX Server (1003735)
    
    
• Ensure that the host field within the /etc/nsswitch.conf file is set to hosts:files. Reboot the host for the changes to take effect.
  
  Note: If using DNS for authentication, ensure that the hosts field is set to hosts:dns files.
  
  
  

Restart the Management agents on the  ESXi/ESX Server and add the host to vCenter Server. Restarting the Management agents on an ESX Server (1003490).