ESXi hosts report 'Partitioned' or 'FDMUnreachable' during physical network congestion

Products

• VMware vSphere ESXi

Issue/Introduction

You observe the following symptoms across multiple ESXi hosts or independent vCenter environments:

• Virtual Machines (VMs) are unexpectedly rebooted by vSphere HA.
• vCenter Server reports: vSphere HA failover operation in progress.
• ESXi hosts show status as Partitioned or FDMUnreachable.
• NSX-T/VCF environments report BFD (Bidirectional Forwarding Detection) overlay flaps.

Environment

vSphere HA
ESX 8.x
vCenter 8.x
NSX 4.x

Cause

This issue occurs when a transient, external physical network disruption—such as a packet storm or severe network congestion—simultaneously isolates ESXi hosts from their management gateways and storage heartbeat datastores.

Based on the fdm.log and physical switch statistics, the VMware HA mechanism functions as designed by detecting the loss of communication and initiating failover. High discard or throttle statistics on physical switches (e.g., Dell, Cisco) typically correlate with these events.

Resolution

There is no configuration change required within the VMware software layer. To resolve and prevent recurrence, investigate the physical network infrastructure:

1. Verify Physical Switch Health: Check for high drop/discard/throttle statistics on ports connected to ESXi hosts and storage arrays.
   • Potential causes of a transient packet storm: Spanning Tree Protocol (STP) topology changes, microbursts overwhelming switch port buffers, or routing loops. Engage the physical network administrator team or switch vendor.
2. Review Host Logs: Search /var/run/log/fdm.log for the following patterns:
   • Host #### changed state: FDMUnreachable
   • Host #### changed state: Partitioned
   • Datastore (/vmfs/volumes/<UUID>) is inaccessible to host-####.
   • Correlate the exact timestamps of the FDMUnreachable and BFD flap events with the physical switch discard/throttle events. Note: Log timestamps are in UTC.
3. Check NSX BFD Status: Review /var/run/log/vmkernel.log for overlay flaps indicating control detection time expiration:
   • local: ####, remote: ####, oldState: up, newState: down, diag: Control Detection Time Expired
4. Check Network-based Storage Status: Review /var/run/log/vmkernel.log for the following types of messages
   
   • All Paths Down (APD), Path Down (PDL), or iSCSI/NFS latency spikes that would confirm the storage network was also congested during the event.
5. Monitor for Self-Resolution: In cases of transient congestion, the issue may resolve organically once the physical layer stabilizes. If the issue re-occurs frequently, consider deploying monitoring tools such as VMware Aria Operations for Networks or setting up packet captures on an ESX host (pktcap-uw) during the next occurrence.

Additional Information

• Determining if your vSphere HA cluster has experienced a host failure - This is a general, foundational guide that explains how to read the fdm.log to identify if a host was partitioned, isolated, or dead.
• vSphere HA Failover Triggered by Sustained Physical NIC Flapping - This KB discusses vSphere HA failovers due to physical network maintenance where the actual physical link (vmnic) goes fully DOWN and UP. 
• BFD and BGP Flaps on NSX Edge due to Underlay Packet Loss - This article focuses on the NSX Edge node and BGP routing resets due to dropped underlay packets (Control Detection Time Expired).
• Troubleshooting NSX TEP/BFD Tunnels between ESXi hosts and Edges - This is a standard NSX troubleshooting guide for BFD tunnels going down. It focuses on persistent configuration issues like MTU mismatches, blocked firewall ports, and routing errors.