Testing jumbo frame pings from ESXi over the network

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

• vMotion failures may be observed
• TLS traffic may not be seen on the host, such as when doing "openssl s_client -connect myhost:8182" from an affected host, the tcp handshake happens but no TLS traffic is seen.
• ESXi host(s) may show as unavailable for vSphere HA purposes
• You may see errors related to NFS, FCoE, or software iSCSI storage
• You may see errors with components where it seems network issues may be causing failures to send information between ESXi hosts or vCenter Servers

Environment

VMware ESXi 7.0.x
VMware ESXi 8.0.x

Cause

There are many types of traffic, most notably vMotion and storage communication, which have higher throughput when passed with a Maximum Transmission Unit (MTU) setting of 9000, otherwise known as "jumbo" frames. An MTU setting of 1500 is considered a "standard" frame.

When network frame size is set to jumbo frames in an ESXi host's vmkernel NIC configuration, those frames might be dropped or fragmented by any other device that is set to only pass standard sized frames. Therefore, the frame size setting for all devices along the network path must allow jumbo frames to pass in order for the communication to work.

Devices along the network path could be virtual switch ports, physical switch ports, switches (such as blade chassis switches), routers, virtual switches, vmks, or vmnics of other ESXi hosts.

Having a higher frame size on physical hardware but not on an ESXi's vmkernel NICs will not cause a problem. Standard frame packets can always pass through jumbo frame-configured devices.

Resolution

Please note that there could be some vmks, vmnics or virtual switches configured to use standard frames, such as for management traffic, while there are other vmks, vmnics or virtual switches configured for jumbo frame traffic, such as for storage and vMotion.

First, check that the connectivity concern is over VLAN connections, not VXLAN or GENEVE

• VXLAN and GENEVE are related to NSX

Ensure all ESXi host MTU settings for the desired jumbo frame networks are consistently set to the same MTU size. The following examples assume an MTU of 9000.

You can do this through the vCenter Server user interface per Enabling Jumbo Frames on virtual switches. To do this through the command line interface (CLI):

1. Connect to the ESXi host console to access the CLI
2. Display the virtual switch MTU settings
   • # esxcfg-vswitch -l
3. Display the vmk MTU settings
   • # esxcfg-vmknic -l
4. Display the vmnic MTU settings
   • # localcli network nic list
5. You'll usually be testing ESXi host to ESXi host traffic, so connect to these other host(s) and check their settings as well. ​

Test whether the MTU settings are the same throughout the physical network path as well

1. Note the vmk numbers of all vmks that need to be tested
2. Connect to the ESXi host console to access the CLI
3. For each vmk and destination device, run a command similar to the following:
   • # vmkping -I vmk<x> <destination_IP> -d -s 8972
   • If testing iSCSI destinations, this would usually involve two iSCSI destinations
   • If testing vMotion, you would usually test all other ESXi hosts in same cluster as the current host.
   • In all of these commands, for testing a particular stack such as vMotion, you can add a string to only test that stack. Example:
     • # vmkping -I vmk<x> -S vmotion <destination_IP> -d -s 8972
4. You can also test if a standard MTU size of 1500 is working,
   • # vmkping -I vmk<x> <destination_IP> -d -s 1472
5. You can perform additional tests to see if the physical upstream switchport is not configured for jumbo frames: 
   • # vmkping -I vmk<x> <destination_IP> -d -s 1473
   • If the command fails
     • The physical switch MTU value is likely still at the 1500 default value
   • If this command worked but the test for an MTU of 8972 failed
     • An MTU of 9000 might have been set at the switch but didn't allow for correct overhead
       • Overhead differs for different physical switch types
       • Check with your network device vendor for the correct overhead value

Changing MTU settings (standard switch only)

You can do this through the vCenter Server user interface per Enabling Jumbo Frames on virtual switches. To do this through the command line interface (CLI):

1. For each affected standard vSwitch:
   • # esxcfg-vswitch -m <MTU-1500_or_9000> <switchname>
2. For each affected vmk:
   • # esxcfg-vmknic -m <MTU-1500_or_9000> vmk<x>
3. The vmnics will inherit the MTU setting from the switch settings and don't need to be set separately.

 

Additional Information

For distributed switches, see What are Jumbo Frames