vSAN Cluster Partition due to MTU Mismatch or Network Misconfiguration

Products

• VMware vSAN

Issue/Introduction

A vSAN cluster reports multiple critical health alerts in Skyline Health, primarily centered around a vSAN Cluster partition and vSAN: Basic (unicast) connectivity check. In two-node ESA configurations, these warnings are frequently caused by MTU mismatches between data nodes and the Witness Appliance, potentially leading to isolated hosts and unresponsive virtual machines.

Symptoms

• Skyline Health reports failures for vMotion MTU check and vSAN MTU check (ping with large packet sizes).
  
• vSAN Skyline Health displays red alerts for Overall health findings (Network misconfiguration) and vSAN cluster partition.
• esxcli vsan cluster get shows a Sub-Cluster Member Count of 1, indicating the host is isolated.
• Standard vmkping attempts fail with 100% packet loss or sendto() failed (Host is down) socket errors.
• Testing connectivity to the Witness Appliance with vmkping -I vmkX -d -s 8972 fails despite successful pings between data nodes.
• vCenter Server is not reachable and virtual machines are not responding.
• Unable to power on virtual machines on the affected host.
• Newly added ESXi host does not appear as part of the vSAN cluster.
• esxtop output displays entries with "void" for the assigned vSAN vmnic.

Environment

• VMware vSAN (All Versions)
• Two-node vSAN cluster or Stretched Cluster configurations
• Distributed Virtual Switch (VDS) using Jumbo Frames (MTU 9000)

Cause

The partition is caused by a networking misconfiguration where the vSAN traffic IP/VMkernel adapter is tagged with an incorrect VLAN, or an architectural MTU mismatch exists. Additionally, firewall or switch rules may be blocking required ports (TCP 2233 and UDP 12321) for vSAN heartbeats and communication. This can be triggered on an existing cluster after switch upgrades, configuration changes, or migrations of hosts to new switches.

Resolution

Address the underlying cause of the partition:

1. Audit physical switch ports connected to data nodes and the Witness Appliance to ensure identical VLAN tagging (IEEE 802.1Q).
2. Verify that TCP Port 2233 and UDP Port 12321 are open and stable across the entire network path between data nodes and the Witness.
3. Identify the active VMkernel interfaces and verify their assigned vSAN traffic types by running: esxcli vsan network list
4. Identify the vSAN VMkernel Interface configuration by running esxcfg-vmknic -l.
5. Standardize the Cluster MTU Architecture:
   • Update the Witness vmk MTU to match data nodes: esxcli network ip interface set -i vmkX -m 9000.
   • Alternatively, implement Witness Traffic Separation (WTS) using a dedicated MTU 1500 VMkernel interface if the physical network cannot support MTU 9000.

Validate the problem has been resolved:

1. Validate end-to-end unicast and jumbo frame connectivity using vmkping -I vmkX -d -s 8972 to ensure 0% packet loss.
2. Verify the cluster status with esxcli vsan cluster get to confirm all members are present.

Additional Information

• For enhancements or defect updates, subscribe to this article.
• Refer to KB 326579 for additional scenarios involving vSAN node isolation.
• Troubleshooting vSAN Networking.
• Contact Broadcom Support (Link).
• vSAN Network Port Requirements
• https://ports.broadcom.com/view/?vcfVersion=9.0+vSAN