vSAN MTU Check (Ping with Large Packet Size) Skyline Health Errors

Products

VMware vSAN

Issue/Introduction

vSAN clusters may report multiple critical health alerts in Skyline Health, primarily regarding the "vSAN: MTU check (ping with large packet size)" test. These alerts frequently indicate a network configuration mismatch, often occurring after physical switch migrations, trunk modifications, or host re-installations, leading to cluster partitions or object inaccessibility.

Symptoms

Skyline Health Alerts: "vSAN: MTU check (ping with large packet size)" and "vMotion: MTU check".
Cluster Health: vSAN cluster reported as "Partitioned" or "Network Partition".
Object Health: Objects on vSAN datastore display as "Inaccessible".
Performance: Severe performance degradation on vSAN datastore.
VM/Guest Status: Guests become non-responsive, fail to power on, or go into an inaccessible state.

Verification:

To validate the faulty VMK and host, click on the Troubleshoot option for the vSAN: MTU Check (Ping with Large Packet Size) error.

or
Based on the screenshot below, it is confirmed that Host 1 is unable to communicate with the other two hosts in the cluster via VMK1 with an MTU of 9000 as Host 1 check to the other hosts are failing, and the checks from all other hosts to Host 1 are failing.
In case of a stretched cluster (2 node + witness), 'vSAN MTU Ping check failure' (Ping with Larget Packet Size) may also be reported even when data nodes communicate to witness nodes over 1500 MTU. There may be ping drops between data nodes and witness node in such cases for 8972 payload.

Environment

VMware vSAN 7.x
VMware vSAN 8.x

Cause

The issue is caused by an MTU mismatch between the ESXi hosts (expecting Jumbo Frames/MTU 9000) and the physical network fabric (restricted to standard MTU/1500). When the source does not fragment packets and the physical switch drops them, cluster communication breaks down. This can be triggered by several activities including:

New cluster creation where physical infrastructure does not match the host configurations
Migrating host physical networking to new switches
Performing switch work resulting in changes to port configurations
Physical network infrastructure changes such as changing ISLs or trunking

This can be validated by the following steps:

Run the command "esxcli vsan network list" to identify the VMK used for vSAN traffic
'esxcli vsan network list'
Interface
VmkNic Name: vmk1
IP Protocol: IP
Interface UUID: ########-####-####-####-########
Agent Group Multicast Address: ###.#.#.#
Agent Group IPv6 Multicast Address: ####: :#:#:#
Agent Group Multicast Port: ####
Master Group Multicast Address: ###.#.#.#
Master Group IPv6 Multicast Address: ####: :#:#:#
Master Group Multicast Port: #####
Host Unicast Channel Bound Port: #####
Data-in-Transit Encryption Key Exchange Port: 0
Multicast TTL: 5
Traffic Type: vsan

In the above example, it is confirmed that vmk1 is used for vSAN traffic.

Run the command "esxcfg-vswitch -l" to identify the vSwitch used for vSAN traffic and check the MTU configured on it.
esxcfg-vswitch -l
DVS Name Num Ports Used Ports Configured Ports MTU MTU
Switch name 2520 10 512 9000
DVPort ID In Use Client
### 1 vmnicl
### 1 vmnic0
### 0
### 0
# 1 vmk0
### 1 vmk1
### 1 vmk2

In the above example, the vSAN VMkernel adapter (vmk1) is associated with the specified vSphere Distributed Switch (vDS). Therefore, it is confirmed that vmnic1 and vmnic0 are being used for vSAN communication and the vSwitch is configured with an MTU of 9000.
Run the command "esxcfg-vmknic -l" to verify the MTU set on the VMkernel adapter (vmk)
esxcfg-vmknic -l
vmk1 128 IPv4 9000
65535 true STATIC DefaultTCPIPStack

In the above example, it is confirmed that vmk1 is configured with MTU 9000.
.Run the command "esxcfg-nics -l" to confirm the MTU configured on the physical nics (vmnics).
esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnico ####: ##: ##: # vmxnet Up 10000Mbps Full ##:##:##:##:##:##:#### 9000
vmnicl ####: ##: ##: # vmxnet Up 10000Mbps Full ##:##:##:##:##:##:#### 9000

In the above example, it is confirmed that vmnics are configured with MTU 9000.

Note: Repeat the above procedure for all the hosts in the cluster and make sure the MTU should be consistent across the network. It's important to ensure that the MTU setting is consistent across the entire environment — including the vSphere VMkernel interfaces, VMNICs, and the physical switch ports. In some cases, MTU mismatches can occur even within vSphere itself, between the VMkernel and the VMNICs..
Ping the faulty host from a working host using a 9000 MTU is not working.
'vmkping -I <vmkX> -d -s 8972 <Destination_IP>'
PING ##.##.###.## ( ##.##.###.##): 8972 data bytes

--- ##.##.###.## ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
Ping the faulty host from a working host using a 1500 MTU is working fine.
vmkping -I vmkx -d -s 1472 <IP adress of faulty node>"

PING ##.##.###.## (##.##.###.##): 1472 data bytes
1480 bytes from ##.##.###.##: icmp_seq=0 ttl=64 time=0.118 ms
1480 bytes from ##.##.###.##: icmp_seq=1 ttl=64 time=0.116 ms
1480 bytes from ##.##.###.##: icmp_seq=2 ttl=64 time=0.106 ms

--- ##.##.###.## ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.106/0.113/0.118 ms

Based on the above results, it shows that vSAN communication fails between the healthy and faulty hosts with MTU 9000, but works with MTU 1500. This points to an MTU mismatch somewhere in the environment.

The VMkernel adapter on the Host has the MTU of 9000, but the physical switch enforces the MTU of 1500.
The failure occurs because the source does not fragment the packet and the physical switch drops the packet.

Resolution

If there is an MTU mismatch within vSphere between the VMkernel and the vmnics, please change the MTU value.

For example : If the correct MTU for the environment is 1500 and the VMK is set 9000, change it to 1500 to allow for cluster creation.

- To configure Jumbo Frames on a vDS in vSphere Web Client:

1. 1. Browse to a distributed switch in the vSphere Web Client navigator.
  2. Click the Actions tab, and click Settings > Edit Settings.
  3. Click Advanced and set the MTU property to a value greater than 1500 bytes.
    - You cannot set the MTU size to a value greater than 9000 bytes.
    - When changing the MTU size in a vDS, the attached uplinks (physical NICs) are brought down and up again. This causes a short network outage for virtual machines and/or services that are using the uplinks.
  4. Click OK.

- To configure Jumbo Frames on a vSS in vSphere Web Client:

1. In the vSphere Web Client, navigate to the host.
2. On the Configure tab, click Virtual Switches.
3. Navigate to the virtual switch, then click Edit.
4. Set the MTU value to 1500.

- To enable Jumbo Frames on a VMkernel port using the vSphere Web Client in vCenter Server:

1. In the vSphere Web Client, navigate to the host.
2. On the Configure tab, click VMkernel Adapters.
3. Click Edit.
4. Set the MTU value to 1500.

Note: MTU size can be increased up to 9000 bytes.

Refer Enabling Jumbo Frames on virtual switches

If the MTU settings within vSphere are correct, please engage the switch or network vendor to verify the MTU configuration on all external network components and ensure they are set to correct value.
In case of MTU 1500 used for communication between data nodes and Witness node (in Stretched cluster scenarios), this alert of MTU check ping failure can be ignored and alert be silenced.