vSAN -- Host is severely degrading vSAN Performance / vSAN Health Alarm: vSAN: MTU check (ping with large packet size)

Products

• VMware vSAN
• VMware vSAN 7.x
• VMware vSAN 8.x

Issue/Introduction

One or more of the following Symptoms are observed in regard to one particular Host:

 

• Network Performance Test: Warning

Example Screenshot:

 

 

• vSAN Health: Warning "MTU check (ping with large packet size)" in regards to vmk(s) used for vMotion and/or vSAN Network  

Example Screenshot:

 

 

• vSAN Health: Warning "Network latency check" 

Example Screenshot:

 

 

• Packet Loss on the vSAN and/or vMotion Network:

When conducting ping tests with the configured MTU between the affected Host and other Hosts in the Cluster,

via their vSAN and/or vMotion VMkernel ports you observe packet loss of up to 100%.

 

Note:

To identify the VMkernel port used for vSAN and/or vMotion Traffic log into the Host via SSH/Putty and run these commands:

esxcli vsan network list

esxcfg-vmknic -l

 

 

Example Output:

 

 

Environment

vSAN 7.x

vSAN 8.x

 

Cause

Issue on the physical Network Layer in relation to errors being reported on the related Network card(s) used for vSAN and/or vMotion Traffic.

1.) On the affected Host, identify the related Network card(s) via ESXTOP

 

1.1)   Start ESXTOP via:

[root@localhost:~] esxtop

1.2)   As soon as the Screen has loaded, switch to the Network Section by hitting "n"

Example Output of the Network Section showing the currently used vmnics (= Network cards):

 

 

2.) For the identified vmnic(s) from previous step: vmnic stats show counters > 0 for "dropped" and/or "errors"

 

Example Output

 

[root@host:~] esxcli network nic stats get -n vmnic##

 

NIC statistics for vmnic##:

   Packets received: 218942955

   Packets sent: 112585880

   Bytes received: 996740989446

   Bytes sent: 196166199720

   Receive packets dropped: 0

   Transmit packets dropped: 0

   Multicast packets received: 18037

   Broadcast packets received: 84908442

   Multicast packets sent: 745

   Broadcast packets sent: 77141

   Total receive errors: 109

   Receive length errors: 0

   Receive over errors: 0

   Receive CRC errors: 109

   Receive frame errors: 0

   Receive FIFO errors: 0

   Receive missed errors: 0

   Total transmit errors: 0

   Transmit aborted errors: 0

   Transmit carrier errors: 0

   Transmit FIFO errors: 0

   Transmit heartbeat errors: 0

   Transmit window errors: 0

 

Resolution

Engage Network Vendor for further investigation of physical Network Layer issue (e.g. SFP, Cabling, Network cards etc)

Additional Information

In regards to MTU Mismatch/Misconfigure see KB 391812