ESXi host disconnects from vCenter Server after adding or connecting it to the inventory

Products

VMware vSphere ESXi

Issue/Introduction

This article helps identify issues with heartbeat traffic between vCenter Server and ESXi .

The steps provided in this article and the included Knowledge Base article links help determine if heartbeat packets are being dropped or lost.

Symptoms:

An ESXi host disconnects from vCenter Server.
After adding or reconnecting an ESXi to the vCenter Server inventory, it disconnects 30 to 90 seconds after the task completes.
Changing the uplink switch port VLAN information for the new IP of the ESXi host before changing the IP of the ESXi host results in the host showing as disconnected in vCenter Server.
Changing the IP address of the ESXi host using the DCUI without first removing the host from the vCenter Server inventory results in the host showing up as disconnected in vCenter Server.

Environment

VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.0
VMware vSphere ESXi 7.x

Cause

This issue occurs when heartbeat packets are dropped, blocked, or lost between the vCenter Server and the ESXi .

It is important to remember that the default heartbeat port is UDP 902, and these packets must be sent by the ESXi to vCenter Server for the ESXi to stay connected and remain in the vCenter Server inventory.

Resolution

To troubleshoot this issue, ensure that bidirectional heartbeat communications are functioning correctly.

The default port for this communication is UDP 902, but be sure to verify the configured port in the vpxa.cfg file on the host. This file also defines the IP address, which manages the host.

Confirm vCenter Server managed IP address

Confirm the vCenter Server managed IP address continuity throughout the environment.

Determine the managed IP address of the vCenter Server:
1. Connect to vCenter Server with the vSphere Client.
2. Click Administration > vCenter Server Settings > Advanced Settings.
3. Make a note of the IP address in the ManagedIP row.
Determine the IP address configured for vCenter Server:

For vCenter Server installed on a Windows Server:
1. From a console or RDP session to the vCenter Server desktop, open a command prompt.
2. Run the command:
  
  ipconfig
3. Make a note of the IP address and ensure that it matches the managed IP address found in step 1.
For vCenter Server Appliance:
1. From a console or SSH session to the vCenter Server Appliance, open a shell prompt. For more information, see Opening a command or shell prompt (1003892).
  
  Note: From the console of the vCenter Server Appliance, press enter on Login.
2. Run the command:
  
  ifconfig
3. Make a note of the IP address next to inet addr: and ensure that it matches the managed IP address found in step 1.
Determine the IP address and port that the ESXi host is using for heartbeat traffic:
1. Connect to the same host using SSH.
2. Check the vpxa.cfg file for the heartbeat traffic port by running the command:
  - On ESXi 6.x:
    
    grep -i serverport /etc/vmware/vpxa/vpxa.cfg
  - On ESXi 7.0U3+:
    
    configstorecli config current get -c esx -g services -k vpxa_solution_user_config |grep -i server_port
3. Ensure that the port number matches the default heartbeat port of 902.
4. Check the vpxa.cfg file for the managed IP address by running the command:
  - On ESXi 6. x:
    
    grep -i serverIp /etc/vmware/vpxa/vpxa.cfg
  - On ESXi 7.0U3+:
    
    configstorecli config current get -c esx -g services -k vpxa_solution_user_config |grep -i server_ip
5. Ensure that the IP address matches the managed IP address found in Step 1.
  
  Note: If the IP address is not the same as the one noted in Step 1, see vCenter Server IP address change causes ESX hosts to disconnect (1001493).

Connectivity

Test connectivity between vCenter Server and the ESXi host through the heartbeat network.

Using telnet and netcat , test connectivity over the heartbeat network through the configured heartbeat port (default 902). For more information, see:

Congestion

Test network congestion:

To test whether UDP heartbeat packets are being received by vCenter Server within the 60-second timeout period, see ESXi/ESX host disconnects from vCenter Server 60 seconds after connecting (1029919).
If packets are delayed due to network congestion, see ESXi/ESX host keeps disconnecting and reconnecting when heartbeats are not received by vCenter Server (1005757).
If NAT is configured, see ESXi hosts connected to vCenter Server using NAT disconnect after 30 to 60 seconds (2019865).

Other troubleshooting areas

Check the NIC firmware and driver levels on the affected host and compare these with other hosts. For more information, see Determining Network/Storage firmware and driver version in ESXi
Capture a network trace and analyze for inconsistency. For more information, see Using the pktcap-uw tool in ESXi 5.5 and later (2051814)

Additional Information

For more information on similar issues, see:

VMware vCenter Server IP address change causes VMware ESXi hosts to disconnect
ESXi hosts enter a Not Responding state after connecting to vCenter Server
Opening a command or shell prompt
ESXi host disconnects intermittently from vCenter Server
Verifying the VMware vCenter Server Managed IP Address
TCP and UDP Ports required to access VMware vCenter Server, VMware ESXi and ESX hosts, and other network components
Determining Network/Storage firmware and driver version in ESXi
ESXi host disconnects from vCenter Server 60 seconds after connecting
Generating custom or default SSL certificates
ESXi hosts connected to vCenter Server using NAT disconnect after 30 to 60 seconds
Troubleshooting network and TCP/UDP port connectivity issues on ESXi
ESXi randomly disconnect from VMware vCenter Server