Intermittent vSAN Unicast and vMotion MTU Connectivity Failures

Products

• VMware vSAN

Issue/Introduction

Symptoms:

Users may observe the following intermittent health alarms in the VMware Skyline Health for vSAN dashboard:

• vSAN: Basic (unicast) connectivity check
• Network Health - Hosts small ping test (connectivity check)
• vMotion: MTU check (ping with large packet size)
• vSAN Health Service - Hosts disconnected from vCenter Server

Additional secondary symptoms include:

• ESXi hosts intermittently entering a "Not Responding" state in vCenter.
• vMotion migration tasks failing or stalling (often at 21%).
• vSphere HA (High Availability) heartbeat errors reported on the cluster.
• Proactive network performance tests show inconsistent failures across multiple hosts.

Environment

vSAN 7/8.x

Cause

These alarms are typically caused by a malfunction or instability in the physical network infrastructure upstream from the ESXi hosts. Specifically:

1. Switch Fabric Issues: Intermittent packet loss within the physical switch fabric can disrupt the small heartbeat packets used for unicast connectivity and vCenter-to-host communication.
2. MTU Fragmentation: Malfunctioning switch hardware may fail to properly handle jumbo frames (MTU 9000), even if correctly configured, leading to large-packet ping failures.

Resolution

Before proceeding with hardware restarts, validate that the software configuration is correct:

Step 1: Validate Software Configuration

1. Ensure the MTU settings are consistent across the entire path: VMkernel Adapter > Virtual Switch > Physical NIC (vmnic) > Physical Switch Port.
2. Verify that the vSAN and vMotion VMkernel adapters have the correct services enabled.
3. Perform a manual vmkping test from the ESXi CLI to isolate the failure:
   • Small Packet: vmkping -I vmkX <Target_IP> -s 64
   • Large Packet (MTU 9000): vmkping -I vmkX <Target_IP> -d -s 8972

Step 2: Physical Infrastructure Remediation

If the software configuration is verified as correct but the intermittent alarms persist:

1. Engage Networking Team: Review physical switch logs for buffer overflows, ASIC errors, or port flapping.
2. Check for Hardware Errors: Run esxcli network nic stats get -n vmnicX and look for Receive CRC errors. If CRC errors are present, replace the SFP modules or cables.
3. Physical Switch Reboot: In cases where the switch fabric has entered an unstable state, a scheduled reboot of the physical switches serving the cluster is required to restore stable communication.

Impact/Risks

Rebooting physical switches will disrupt network connectivity. Ensure the environment is configured with redundant physical switches (Top-of-Rack) and that the ESXi hosts are utilizing NIC teaming (Active/Standby or Active/Active) to maintain availability during the maintenance window.