vSAN skyline health report error "vSAN: MTU check (ping with large packet size)" due to MTU mismatch
search cancel

vSAN skyline health report error "vSAN: MTU check (ping with large packet size)" due to MTU mismatch

book

Article ID: 317670

calendar_today

Updated On:

Products

VMware vSAN VMware Cloud Foundation

Issue/Introduction

Symptoms:

  • vSAN skyline health report error "vSAN: MTU check (ping with large packet size)".

  • The following error report when upgrading hosts from 9.0.2 to 9.1.

    vSAN health test 'vSAN: MTU check (ping with large packet size)' reported an issue for cluster '##########'. 
    Detailed error message: vSAN reports this issue when the large ping (9000 bytes packet size) pass rate is less than 30% for vSAN network among ESXi host.

Environment

VMware vSAN (All Versions)

 

Cause

This issue occurs due to mismatch in MTUs.

Even when "Witness Traffic Separation" is correctly configured (Data = 9000, Witness = 1500), Skyline health check still attempts a 9000-byte ping, which will fail by design if the WAN doesn't support jumbo frames.

 

Stretched cluster :

  • This issue occurs because the MTU set on the vSAN Witness ESXi host/Appliance network does not match with the vSAN data node network in the vSAN cluster.
  • vSAN Data and Witness traffic shares the same vSAN vmknic, therefore, shares a single MTU setting.

Standard Cluster: 

  • This issue occurs when MTU configured across the network is not uniform.

Resolution

Please note before proceeding:
  • MTU changes require careful planning. An MTU misconfiguration may cause Network disconnects and I/O failures. If needed consult with Network Team.
  • For stability of the vSAN cluster, VMware recommends that the MTU configuration must be consistent in the vSAN network including the WAN & Witness Host/Appliance.

Stretched cluster :

To resolve this issue, evaluate one or more of the following options:
  • Reconfigure the Network used by the vSAN Witness Host to MTU 9000 to match with the MTU configured on the Network used by the vSAN Hosts ( Note: This may not be feasible due to WAN limitations. Consult with Network team )
  • Reconfigure the Network used by the vSAN Hosts to MTU 1500 to match with the Network used to communicate with Witness
  • Separate the Network used to communicate with Witness from the Network used by the Host to communicate to each other (= Witness Traffic Separation)
  • vSAN MTU health check will send large packet size like 9000 to target host, and the actual packet size will exceed 9000 with additional headers.  Hence the packet will be fragmented. But in some scenarios, packet fragmentation is not support by WAN providers. This is a known issue, and we reduce the packet size as 8952 since vSphere 7.0 p01 release. If such health check warning is observed, Run the vmkping command as below on ESXi host.

vmkping -I <target_vmk> <target_IP> -s 8952 -d

As long as vmkping succeeds, there will not be any performance issues and I/O errors. Warning can be safely ignored. For silencing the alert see 

Silencing a vSAN health check

If assistance is required, open a Ticket with VMware by Broadcom Support.

Standard Cluster: 

  • Log into the affected ESXi hosts via SSH and run below mentioned command:

    esxcli vsan network list.

    Note the "VmkNic Name" (example: vmk1).

  • Run the following command using the VMK identified above and the IP of target host in the cluster: 

    vmkping -I vmkX <Target_IP> -d -s 8972

  • If the ping fails, there is an MTU mismatch.
  • Engage networking tean and ensure that Jumbo Frames (MTU 9000 or higher) are enabled end to end on all physical switches and ports carrying vSAN traffic.

 

Additional Information

 

 
Powered by Wolken Software