ESXi host disconnects intermittently from vCenter Server

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptom:

• ESXi hosts frequently disconnect from vCenter Server.

• In the vSphere Client, affected hosts appear as "Not Responding".

• vCenter Server is intermittently failing to receive heartbeat packets from the ESXi hosts.

• The vCenter Server log file (/var/log/vmware/vpxd/vpxd.log) contains entries similar to the following:

   

  2025-08-06T21:03:27.731Z [VpxdIntHost] Missed 2 heartbeats for host esx.example.com
  2025-08-06T21:03:27.731Z info vpxd[709621] [Originator@6876 sub=HostCnx opID=CheckforMissingHeartbeats-4b15b2a7] [VpxdHostCnx] No heartbeats received from host; cnx: 52066c40-####-####-####-96df6ec544cb, h: host-908###, time since last heartbeat: 6745344ms
2025-08-06T21:03:27.731Z info vpxd[709621] [Originator@6876 sub=HostCnx opID=CheckforMissingHeartbeats-4b15b2a7] [VpxdHostCnx] No heartbeats received from host; cnx: 528b7944-####-####-####-14cf56852fd2, h: host-910###, time since last heartbeat: 12999144ms
2025-08-06T21:03:27.731Z info vpxd[709621] [Originator@6876 sub=HostCnx opID=CheckforMissingHeartbeats-4b15b2a7] Marking the connection alive to false: 528b7944-####-####-####-14cf56852fd2
2025-08-06T21:03:27.732Z info vpxd[709621] [Originator@6876 sub=InvtHostCnx opID=CheckforMissingHeartbeats-4b15b2a7] Got lost connection callback for host-910####
  
  
  

         This behavior typically indicates intermittent loss of UDP heartbeat traffic between the ESXi hosts and the vCenter Server, often due to network congestion, packet drops, or firewall configuration issues.

Environment

VMware vCenter Server

VMware vSphere ESXi

Cause

• This issue arises when vCenter Server fails to receive the UDP heartbeat messages sent by an ESXi host. ESXi hosts transmit these heartbeats every 10 seconds, and vCenter expects to receive at least one within a 60-second window. If no heartbeat is received during this period, the host is marked as "not responding." This behavior may indicate network congestion, packet loss, or a misconfigured firewall between the ESXi host and the vCenter Server.

Note: If the host consistently disconnects at 60-second intervals, it is a strong indication that UDP port 902 traffic from the ESXi host to the vCenter Server is being blocked most commonly by a firewall.

Resolution

Validating ESXi Heartbeat Communication to vCenter

To confirm that the ESXi host is sending heartbeat packets to the vCenter Server every 10 seconds over UDP port 902, perform the following checks.

Step 1: Verify Heartbeat Transmission from ESXi Host

From an SSH session on the ESXi host, run the following command:

esxi#pktcap-uw --uplink vmnicx --capture UplinkSndKernel --udpport 902 -o -| tcpdump-uw -enr -

• This confirms that the ESXi host is actively sending heartbeat packets to the specified vCenter Server.

Step 2: Verify Heartbeat Reception on vCenter Server

From an SSH session on the vCenter Server Appliance (VCSA), run the following command:

vcsa# tcpdump src host <esxi_host_ip_address> and udp port 902

• This confirms that the vCenter Server is receiving heartbeat packets from the ESXi host.

Next Steps Based on Test Results

• Heartbeats sent but not received:
  Investigate the network path between the ESXi host and vCenter for potential issues such as firewalls, ACLs, or other traffic filtering mechanisms blocking UDP port 902.

• Heartbeats not sent from ESXi:
  Inspect ESXi host services and relevant log files to identify and resolve the root cause.

• Heartbeats sent and received successfully:
  If heartbeat packets are reaching vCenter but hosts are still disconnecting, the issue is likely not network-related. Further investigation into the vCenter Server itself is recommended.

Temporary Workaround: Adjust Heartbeat Timeout

As a short-term measure, increase the heartbeat timeout value in vCenter Server to allow more time for ESXi host responses.

Steps to Update Timeout via vSphere Client:

1. Open the vSphere UI in a web browser and log in.

2. Select the vCenter object from the Hosts and Clusters inventory.

3. Navigate to the Configure tab.

4. Under Settings, select Advanced Settings.

5. Click Edit.

6. In the Key field, enter:

   config.vpxd.heartbeat.notRespondingTimeout

7. In the Value field, enter:

   120

   (adjust this value as needed.)

8. Click Add, then OK to save the change.

9. Restart the vCenter Server service by running:

   vcsa# service-control --stop vmware-vpxd && service-control --start vmware-vpxd

Note: Increasing the timeout is a temporary workaround. It is strongly recommended to identify and resolve the underlying network issue.

Additional Information

If the issue persists despite making all the above changes or to troubleshoot on the Network PCAP, please contact Broadcom support-

Creating and managing Broadcom support cases