VMs Running on vSAN hang due to Packet Loss on spine of the physical network

Products

• VMware vSAN

Issue/Introduction

• Virtual Machines (VMs) on a vSAN cluster stop responding to network requests or hang unexpectedly.
  
  
• vCenter/Skyline Health reports the error: "vSAN: MTU check (ping with large packet size)".
  
  
• ESXi host logs (/var/log/vmkernel.log or /var/log/vobd.log) show "Lost Access to volume" messages.
  
  
• Symptoms may recover automatically after a period of time (e.g., 30 minutes) but recur intermittently.

Environment

VMware vSphere ESXi 8.x

Cause

• The issue is caused by instability in the physical network environment. For example, a faulty physical uplink cable between a Top-of-Rack switch and the network spine.
  
  
• vSAN is dependent on a stable and consistent network connectivity.
• Intermittent packet loss, particularly with Jumbo Frames (MTU 9000), can disrupt storage heartbeating and unicast communication between data sites.
  
  
• This disruption causes ESXi to lose access to vSAN volumes, directly leading to VM hangs.

Resolution

• Identify Physical Layer Errors: Check for NIC drops or CRC errors.

Run the following command on affected ESXi hosts to check for physical NIC errors.
Replace vmnic# with the appropriate physical interface used for your vSAN traffic:

esxcli network nic stats get -n vmnic#

 

• Verify MTU and Connectivity: Test the unicast agent path.

Test the connection using a large packet size (Jumbo Frames) to ensure the physical path supports the configured MTU without loss.
Replace vmk# with your vSAN VMkernel adapter and <Destination_IP> with the target vSAN IP:

vmkping -I vmk# <Destination_IP> -s 8972 -d -c 20

Note: The -d flag in this command sets the "Do Not Fragment" bit, which ensures the test accurately validates the full MTU size across the network path.

• Conduct a Physical Hardware Audit: Work with Network Vendor.
  

If the vmkping tests show packet loss or the interface statistics show errors, collaborate with networking vendor to:

• • • Inspect and replace physical uplink cables (DAC/Fiber).

    • Reseat or replace SFP+ modules.

    • Check switch port statistics for CRC errors or PHY symbol errors.

 

• Verify the Fix: Ensure health checks pass.

After replacing any faulty hardware or resolving the physical layer issue, re-run the vmkping command above.

Finally, verify in vCenter that the vSAN Skyline Health MTU Check alarm has successfully cleared.