ESXi hosts repeatedly disconnect and reconnect every minute due to a static route misrouting management traffic across subnets

Products

• VMware vCenter Server

Issue/Introduction

When vCenter Server and ESXi hosts reside on different Layer 3 subnets, managed ESXi hosts repeatedly transition to a "Not Responding" state approximately every 60 seconds, before automatically returning to a "Connected" state.

• Virtual machines running on the affected hosts continue operating normally with no workload impact.
• The issue affects specific hosts on a given management subnet (e.g., 192.168.100.x), while hosts on other subnets managed by the same vCenter remain stable.
• Manual reconnection attempts via the vSphere Client succeed initially, but the disconnect cycle resumes after 60 seconds.
• vpxd.log on vCenter Server repeatedly reports missed heartbeats and entitlement errors:

info vpxd[#####] [Originator@6876 sub=HostCnx opID=CheckforMissingHeartbeats-#####] [VpxdHostCnx] No heartbeats received from host; cnx: <UUID>, h: host-###, time since last heartbeat: 66000ms
warning vpxd[#####] [Originator@6876 sub=InvtHostCnx opID=HostSync-host-###-#####] Connection not alive due to missing heartbeats; [vim.HostSystem:host-###,<ESXi_Host_FQDN>]
error vpxd[#####] [Originator@6876 sub=AuthorizeManager] com.vmware.esx.authentication.client_profiles.unknown_entitlement Scope: HostSystem:host-###, Privilege: Global.Licenses

• vpxd-profiler.log on vCenter Server displays an uninitialized heartbeat timestamp for the affected host:

--> /HostStatus/HostId/host-###/HBInfo/IP <ESXi_Host_IP>
--> /HostStatus/HostId/host-###/HBInfo/LastHbTime 1970-01-01T00:00:00Z

Running a packet capture on the primary management interface (vmk0) shows no outbound UDP port 902 heartbeat traffic destined for the vCenter IP. (tcpdump-uw -i vmk0 dst host <vCenter_IP> and udp port 902 -nn)

Environment

vCenter Server

Cause

This issue occurs when an explicit static IPv4 route configured on a non-management VMkernel interface (e.g., vmk1) encompasses or overlaps with the vCenter Server IP subnet (<vCenter_Subnet>/24).

1. Routing Intercept: When vpxa on the ESXi host generates UDP 902 heartbeat packets destined for the vCenter Server IP (<vCenter_IP>), the ESXi kernel evaluates the routing table.
2. Interface Misrouting: Because <vCenter_IP> matches the specific static route pointing out of vmk1 (e.g., vSAN or Replication network), the kernel forces the management/heartbeat packets out of vmk1 instead of vmk0 (Management Network).
3. Packet Drop: The gateway configured on vmk1 drops or misroutes the UDP 902 packets because vmk1 belongs to an isolated VLAN/subnet.
4. Heartbeat Loss: Because vCenter never receives UDP 902 heartbeats, LastHbTime remains set to 1970-01-01T00:00:00Z. Exactly every 60 seconds, vCenter marks the host as "Not Responding" and triggers a reconnect cycle.

Resolution

Identify and remove the overlapping static route from the affected ESXi host to allow management traffic to route natively over vmk0.

Step 1: Identify the Misconfigured Static Route
Log into the affected ESXi host via SSH as root and inspect the active routing table:

esxcli network ip route ipv4 list

Example Output:

Network          Netmask          Gateway         Interface
-------          -------          -------         ---------
default          0.0.0.0          192.XX.XX.X   vmk0
192.XX.XX.X     255.255.255.0    192.XX.XXX.X   vmk1  <-- MISCONFIGURED OVERLAPPING ROUTE

Confirm that the vCenter Server IP subnet (<vCenter_Subnet>/24) is bound to a non-management interface (vmk1).

Step 2: Remove the Overlapping Static Route
Remove the route targeting the vCenter Server subnet from vmk1: (Use KVM console and login to the ESXi host)

esxcli network ip route ipv4 remove -n <vCenter_Subnet>/24 -g <vSAN_or_NonMgmt_Gateway_IP>

Step 3: Verify Routing and Outbound Heartbeats

esxcli network ip route ipv4 list

Verify that outbound UDP 902 heartbeat traffic is now flowing out of vmk0:

tcpdump-uw -i vmk0 dst host <vCenter_IP> and udp port 902 -nn

Packets will appear leaving vmk0 every 10 seconds.

• Monitor the host connection status in the vSphere Client. The host will remain in a stable Connected state without dropping off every minute.