Testing jumbo frame pings from ESXi over the network

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• vMotion errors
• TLS traffic is not seen on the host, such as, when doing "openssl s_client -connect myhost:8182" from affected, the tcp handshake happens but no TLS traffic is seen.
• Host keeps showing unavailable for vSphere HA purposes
• Network-based storage errors. That is, errors for NFS, FCoE, or software iSCSI storage
• Any other errors where it seems network is frequently failing to send some information between hosts or vCenters

Environment

VMware ESXi 7.0.x
VMware ESXi 8.0.x

Cause

There are many types of traffic, most notably vMotion and storage communication, which has higher throughput when passed with a Maximum Transmission Unit (MTU) setting of 9000, otherwise known as "jumbo" frames. An MTU setting of 1500 is considered a "standard" frame.

When network frame size is set to jumbo frames in an ESXi host's vmkernel NIC configuration, those frames might be dropped or fragmented by any other device that is set to only pass standard frames. Therefore the frame size setting for all devices along the network path must allow jumbo frames to pass in order for the communication to work.

Devices along the network path could be virtual switch ports, physical switch ports, switches (such as blade chassis switches) or routers, or the virtual switches, vmks, and vmnics of other ESXi hosts.

Having a higher frame size on physical hardware but not on ESXi's vmkernel NICs will not cause a problem. Standard frame packets can always pass through jumbo frame-configured devices.

Resolution

Please note that you could have some vmks, vmnics and virtual switches configured to use standard frames, such as for management traffic, but also have other vmks, vmnics and virtual switches configured for jumbo frame traffic, such as for storage and vMotion.

First ensure connectivity concern is over VLAN connections, not VXLAN or GENEVE

• VXLAN and GENEVE are related to NSX
• Always get VMware NSX team to troubleshoot if your issues are related to GENEVE or VXLAN

Ensure all ESXi host MTU settings for the desired jumbo frame networks are consistently set to the same MTU size. The examples below assume an MTU of 9000.

You can do this through the vCenter Server user interface per Enabling Jumbo Frames on virtual switches. To do this through the command line interface (CLI):

1. Connect to the ESXi host console to access the CLI
2. Display the virtual switch MTU settings
   • # esxcfg-vswitch -l
3. Display the vmk MTU settings
   • # esxcfg-vmknic -l
4. Display the vmnic MTU settings
   • # localcli network nic list
5. You'll usually be testing ESXi host to ESXi host traffic, so connect to these other host(s) and check their settings as well. ​

Test whether the MTU settings are the same throughout the physical network path as well

1. Note the vmk numbers of all vmks you need to test
2. Connect to the ESXi host console to access the CLI
3. For each vmk and destination device run command:
   • # vmkping -I vmk<x> <destination_IP> -d -s 8972
   • If testing iSCSI destinations this would usually involve two iSCSI destinations
   • If testing vMotion we usually would test all other ESXi hosts in same cluster as current host.
   • Also in all of these commands, for testing a particular stack such as vMotion you can add a string to only test that stack. Example:
     • # vmkping -I vmk<x> -S vmotion <destination_IP> -d -s 8972
4. You can also test if 1500 MTU is working,
   • # vmkping -I vmk<x> <destination_IP> -d -s 1472
5. Additional tests to see if the physical upstream switchport is not configured for jumbo frames: 
   • # vmkping -I vmk<x> <destination_IP> -d -s 1473
   • If the command fails
     • Physical switch value is still at 1500 default value
     • No one made any changes to switch
   • If this command worked but the test for MTU 8972 failed
     • 9000 MTU was set at the switch but didn't allow for correct overhead
     • Overhead differs for different physical switch types
     • You need to check with your network device vendor for correct overhead value

Changing MTU settings (standard switch only)

You can do this through the vCenter Server user interface per Enabling Jumbo Frames on virtual switches. To do this through the command line interface (CLI):

1. For each affected standard vSwitch:
   • # esxcfg-vswitch -m <MTU-1500_or_9000> <switchname>
2. For each affected vmk:
   • # esxcfg-vmknic -m <MTU-1500_or_9000> vmk<x>
3. The vmnics will inherit the MTU setting from the switch settings so don't need to be set separately.

For distributed switches, please see the documentation located here: What are Jumbo Frames