Stretched vSAN Network alerting for jumbo frames

Products

• VMware vSAN

Issue/Introduction

After a physical cluster move / physical network change you are now seeing the following alert being repotted by vSAN Skyline Health. Between your physical hosts and witness. When utilizing the vSAN witness traffic tag on your management VMK. 

Environment

vSAN (all versions)

Cause

This Alert will be present when you have over 30 percent packet loss when pinging with large packets. 

For example, you can use the MTU of 1500 on the VMkernel adapter and the MTU 9000 on the physical switch. 
If the source VMkernel adapter has an MTU of 1500, it will fragment the 9000 Byte packet. 
The fragments then travel along the network to the other ESXi host where they are reassembled. 
As long as all network devices along the path use a higher or equal MTU, the test passes.

This example below shows that the witness host can ping the physical hosts with a  sub 1000 MTU packet.  But any size packets over 1000 MTU will fail to send causing this alert.  This is due to 100 percent packet accorss the network as a fragmented packet at 8972 MTU will be dropped  due to physical network limitations. 

[ESX01:~] vmkping -I vmk0 ###.###.###.3 -s 1000
PING  ###.###.###.3 ( ###.###.###.3 ): 1000 data bytes

---  ###.###.###.3 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

[ESX01:~] vmkping -I vmk0 ###.###.###.3 -s 972
PING ###.###.###.3 (###.###.###.3): 972 data bytes
980 bytes from ###.###.###.3: icmp_seq=0 ttl=56 time=326.730 ms
980 bytes from ###.###.###.3: icmp_seq=1 ttl=56 time=326.409 ms
980 bytes from ###.###.###.3: icmp_seq=2 ttl=56 time=326.178 ms

---  ###.###.###.3 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 326.178/326.439/326.730 ms

Resolution

Check the physical network to determine what is causing the fragmentation of large sized packets being dropped on the network. 

This issue is gone over in more detail in KB vSAN Healthcheck -- vMotion: MTU check (ping with large packet size)