Large-Scale vCLS VM Re-creations in Production Clusters Following Widespread ESXi Host Heartbeat Loss

Products

• VMware vCenter Server

Issue/Introduction

An environmental monitoring anomaly occurred where a significant number of vCLS virtual machines were unexpectedly destroyed and re-created across production environments. System events indicate that vCenter Server dropped active connections to multiple ESXi hypervisors simultaneously, classifying them in an offline status and triggering automated configuration safeguards. Widespread, parallel failures across numerous separate hosts indicate a centralized network delivery failure rather than an isolated physical host crash.

Symptoms:

 

In vCenter vxpd.log similar lines will be present 

YYYY-MM-DDT HH:MM:S info vpxd[07157] [Originator@6876 sub=HostCnx opID=CheckforMissingHeartbeats-####] [VpxdHostCnx] No heartbeats received from host; cnx: ####, h: Host-####, time since last heartbeat: 68657ms
YYYY-MM-DDT HH:MM:S info vpxd[07157] [Originator@6876 sub=HostCnx opID=CheckforMissingHeartbeats-####] Marking the connection alive to false: ####
YYYY-MM-DDT HH:MM:S info vpxd[07157] [Originator@6876 sub=InvtHostCnx opID=CheckforMissingHeartbeats-####] Got lost connection callback for Host-####
YYYY-MM-DDT HH:MM:S warning vpxd[09142] [Originator@6876 sub=InvtHostCnx opID=HostSync-Host-####] Connection not alive due to missing heartbeats; [vim.HostSystem:Host-####, cnx: ####
YYYY-MM-DDT HH:MM:S info vpxd[09142] [Originator@6876 sub=InvtHostCnx opID=HostSync-Host-####] Skip restoring heartbeat: already got heartbeats; [vim.HostSystem:Host-####, ct: 329750
YYYY-MM-DDT HH:MM:S warning vpxd[09142] [Originator@6876 sub=InvtHostCnx opID=HostSync-Host-####] Exception occurred during host sync; Host communication failed; [vim.HostSystem:Host-####, e: N5Vmomi5Fault17HostCommunication9ExceptionE(Fault cause: vmodl.fault.HostCommunication

Environment

VMware vCenter Server

Cause

vCenter Server service (vpxd) failing to receive UDP Port 902 heartbeat packets emitted from the ESXi hosts within the default 60-second monitoring window due to management plane network congestion or physical network firewall drops.

Resolution

To resolve this issue, perform the following steps:

1. Ensure that UDP Port 902 and TCP Port 443 are permitted for bi-directional communication between the vCenter Server and the ESXi Management network.

2. From the ESXi host SSH session, verify that heartbeats are being sent:
   
   pktcap-uw --uplink <vmnic> --capture UplinkSndKernel --udpport 902 -o -| tcpdump-uw -enr

3. On the vCenter Server (VCSA), verify heartbeats are being received:
   
   tcpdump src host <ESXi_IP_Address> and udp port 902

4. If cluster management state mappings remain desynchronized or entries show as orphaned within the inventory database

5. Log into the vCenter appliance terminal and restart the ESX Agent Manager service:Log into the VCSA via SSH and restart the agent manager:
   
   service-control --restart vmware-eam

1. Reset vCLS Health:

   1. Navigate to the affected Cluster in the vSphere Client.

   2. Retrieve the Cluster Domain ID from the browser URL (e.g., domain-c###).

   3. Navigate to vCenter Server > Configure > Advanced Settings.

   4. Set config.vcls.clusters.domain-c###.enabled to False.

   5. Wait for vCLS VMs to be removed, then set the value back to True.

Additional Information

VCF Cluster Join fails due to vCLS Unhealthy Status caused by Port 902 Heartbeat Loss