"Host Connection and Power State" or "Host Connection Failure" Alarms Triggered on Multiple ESXi Hosts in vCenter

Products

VMware vSphere ESXi VMware vCenter Server

Issue/Introduction

All hosts in vCenter have an alarm triggered for "Host connection and power state":

All hosts and VMs, and vCenter itself, may be currently up and responding normally.
A review of the vCenter Server logs ie journalctl -xe confirms that the connection was lost and quickly restored, triggering automated alarm actions (such as SNMP traps).

YYYY-MM-DDTHH:DD:SS <FQDN_OF_VCENTER> vpxd[41972]: Event [######] [1-1] [YYYY-MM-DDTHH:DD:SS ] [vim.event.AlarmSnmpCompletedEvent] [info] [] [CLUSTER_NAME] [######] [Alarm 'Host connection failure': an SNMP trap for entity <FQDN_OF_ESXI> was sent]
YYYY-MM-DDTHH:DD:SS <FQDN_OF_VCENTER> vpxd[41972]: Event [######] [1-1] [YYYY-MM-DDTHH:DD:SS ] [vim.event.AlarmActionTriggeredEvent] [info] [] [CLUSTER_NAME] [#######] [Alarm 'Host connection failure' on <FQDN_OF_ESXI> triggered an action]
The var/log/vmware/vpxd/vpxd.log reveal accumulation of missed heartbeats, followed by a brief NO_RESPONSE state, and an almost immediate return to CONNECTED:

YYYY-MM-DDTHH:DD:SS info vpxd[42493] [Originator@6876 sub=InvtHostCnx opID=HeartbeatStartHandler-5f###3bdf] Missed heartbeats for host; [vim.HostSystem:host-#####,<FQDN_OF_ESXI>], missed: 1532110, msg: {srv: 1457868, gen: 332837, ct: 1532111, bld: 23307199, cnx: 52####69-6##2-0##9-d4#d-bf5#####8c729, ip: <IP_OF_ESXI>}
YYYY-MM-DDTHH:DD:SS warning vpxd[42110] [Originator@6876 sub=MoHost opID=HB-host-####@62595-443###28] host [vim.HostSystem:host-#####,<FQDN_OF_ESXI>] connection state changed to NO_RESPONSE
YYYY-MM-DDTHH:DD:SS info vpxd[42071] [Originator@6876 sub=MoHost opID=HB-host-###@228-2e###7d7] host [vim.HostSystem:host-#####,<FQDN_OF_ESXI>] connection state changed to CONNECTED

Cause

This issue is caused by a temporary interruption in the UDP heartbeat traffic (Port 902) between the ESXi hosts and the vCenter Server. vCenter tracks host health by expecting a heartbeat every 10 seconds.

There are two primary scenarios that cause this massive accumulation of missed heartbeats:

1. During a vCenter Server upgrade or maintenance reboot, the vpxd service is offline. While offline, the missed heartbeats accumulate. Upon service initialization, if the inventory synchronization takes longer than the allowed grace period, the massive "Missed heartbeats" counter triggers the alarm logic. The hosts returning to a CONNECTED state within seconds confirms this was a processing delay following the outage, rather than an actual host failure.

2. If no vCenter maintenance was performed, this behavior indicates an intermittent loss of UDP heartbeat traffic. Common culprits include:

Network congestion or packet drops.
Firewalls temporarily blocking UDP port 902 traffic.
IP address conflicts (Duplicate IPs) on the vCenter Server or ESXi management networks.

Resolution

The required action depends on the stability of the environment prior to the alarms.

Scenario A: Expected Connectivity Loss

If the alarms occurred immediately following an expected event—such as a vCenter Server upgrade, a deliberate reboot, or a planned network change for the vCenter VM—the alarms are false positives generated during the service recovery phase.

Action: Acknowledge and reset the alarms to green. No further troubleshooting is required.

Scenario B: Unexpected Connectivity Loss

If the vCenter Server was not rebooted or upgraded, the network environment must be investigated to determine why the vCenter temporarily lost connection to the hosts.

Action 1: Verify UDP 902 Ensure that physical and virtual firewalls are not intermittently dropping UDP Port 902 traffic between the ESXi management IP addresses and the vCenter Server.
Action 2: Check for Duplicate IPs: Investigate the network for IP conflicts that may be hijacking traffic intended for the vCenter Server or specific hosts.

Refer ESXi host disconnects intermittently from vCenter Server and Duplicate IP address detected.

Additional Information

Review the following articles for guidance on similar specific conditions

"Host connection and power state" alarm when an ESXi host wakes from standby