vSAN Cluster Partition and Network Instability due to Physical Hardware Failure

Products

• VMware vSAN

Issue/Introduction

A VMware vSAN cluster partition and widespread virtual machine inaccessibility may occur as a result of underlying physical hardware instability, such as Network Interface Card (NIC) faults or memory (DIMM) failures. These hardware disruptions cause excessive packet loss or heartbeat timeouts, forcing ESXi hosts to isolate from the cluster. This article provides the symptoms and identification steps needed to isolate faulty hardware and restore cluster connectivity. Key indicators include cluster fragmentation where the sub-cluster member count is reduced and objects are marked as inaccessible or invalid in the vCenter Server.

 

 

• Multiple virtual machines appear "Inaccessible" or "Invalid."
• vSphere Client (UI) or vCenter Server becomes inaccessible.
• ESXi hosts report a vSAN Network Partition.
• esxcli vsan cluster get displays "Sub-cluster Member Count" of 1 (when multiple nodes exist).
• Physical NICs (vmnic) appear as "void" in esxtop.
• "Wait Interrupted" errors appear in vmkernel logs.

 

Environment

VMware vSAN 7.x, VMware vSAN 8.x

Cause

Physical hardware instability—such as a NIC firmware crash, physical link failure, or memory (DIMM) Machine Check Exception (MCE)—disrupts the vSAN management and data networks. This disruption triggers heartbeat timeouts, forcing the affected ESXi host to leave the cluster. The resulting partition leads to a loss of quorum, causing objects to enter an inaccessible state until the network path is restored or the faulty component is bypassed.

Resolution To clear the network partition and restore object health, the following steps are performed:

Resolution

• Identification of the Faulty Component:
  • Verification of the NIC state using esxtop (Network view) to see if the interface is reported as "void."
  • Review of System Event Logs (SEL) or IPMI logs for hardware alerts such as MCE or DIMM errors.
• Isolation of the Faulty Uplink:
  • Administrative shutdown of the failing interface to force traffic over redundant uplinks: localcli network nic down -n vmnicX
• Connectivity Verification:
  • Connectivity tests between cluster hosts using vmkping to ensure jumbo frame support and low latency: vmkping -I vmkX -d -s 8972 <Destination_IP>
• Hardware Remediation:
  • Coordination with the hardware vendor for replacement of the faulty NIC or DIMM.
• Monitoring:
  • Observation of the vSAN Skyline Health UI for object resynchronization and sub-cluster membership restoration.

 

 

Additional Information

 

• Troubleshooting vSAN Networking
• vSAN Health Service - Network Health
• Contact Broadcom Support