When you have a VMware vSAN Stretched Cluster setup where the witness tag is placed on a non vSAN data tagged vmkernel port (witness traffic separation)
Example: vmk1 is tagged for vSAN traffic while vmk0 is tagged with vSAN witness traffic)
vSAN Network Health Check for MTU check in a Stretched Cluster fails
vSAN Network Health status shows that the Hosts large ping test (MTU check) has failed
vSAN stretched cluster with witness traffic separation enabled
Even without the "do not fragment" flag on the ping test the pings are failing with large packet size (9000 MTU with 8972 size specified). This is caused when actual size of packets allowed to pass through the network link is below 1000.
This can be confirmed by pinging from a data node witness tagged vmkernel port to the witness node vSAN enabled IP which will fail with normal sized packets and succeed with small packet sizes.
vmkping -I vmk# xxx.xxx.xxx.xxx -s 8972vmkping -I vmk# xxx.xxx.xxx.xxx -s 1472vmkping -I vmk# xxx.xxx.xxx.xxx -s 980
Work with your networking team or vendor to find why packet size is restricted across the network and correct the problem.