Test network is passing jumbo network frames between devices

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• vMotion errors
• TLS traffic is not seen on the host, such as, when doing "openssl s_client -connect myhost:8182" from affected, the tcp handshake happens but no TLS traffic is seen.
• Host keeps showing unavailable for vSphere HA purposes
• Network-based storage errors. That is, errors for NFS, FCoE, or software iSCSI storage
• Any other errors where it seems network is frequently failing to send some information between hosts or vCenters

Environment

VMware ESXi 6.5.x
VMware ESXi 6.7.x
VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 8.0.x

Cause

There are many types of traffic, most notably vMotion and storage communication, which is more effective when passed with a Maximum Transmission Unit (MTU) setting of 9000, otherwise known as "jumbo" frames. An MTU setting of 1500 is considered a "standard" frame.

When network frame size is set to jumbo frames in an ESXi host's vmk or vmnic configuration, those frames won't be passed through any other device that is set to only pass standard frames. Therefore frame size setting for all intervening devices must allow jumbo frames to pass in order for the communication to work.

Intervening devices could be virtual switch ports, physical switch ports, switches or routers, or the virtual switches, vmks, and vmnics of other ESXi hosts.

Please note that there is no problem if the physical network is set for jumbo frames but none of the ESXi system is. Standard frame packets can always pass through jumbo frame-configured devices..

Resolution

Please note that you could have some vmks, vmnics and virtual switches configured to use standard frames, such as for management traffic, but also have other vmks, vmnics and virtual switches configured for jumbo frame traffic, such as for storage and vMotion.

First ensure connectivity concern is over VLAN connections, not VXLAN

• VXLAN is related to NSX
• Always get VMware NSX team to troubleshoot if your issues are over VXLAN

Ensure all ESXi host MTU settings for the desired jumbo frame networks are consistently set to 9000.

You can do this through the vCenter Server user interface per Enabling Jumbo Frames on virtual switches. To do this through the command line interface (CLI):

1. Connect to the ESXi host console to access the CLI
2. Display the virtual switch MTU settings
   • # esxcfg-vswitch -l
3. Display the vmk MTU settings
   • # esxcfg-vmknic -l
4. Display the vmnic MTU settings
   • # localcli network nic list
5. You'll usually be testing ESXi host to ESXi host traffic, so connect to these other host(s) and check their settings as well. ​

Test whether the MTU settings are the same throughout the physical network path as well

1. Note the vmk numbers of all vmks you need to test
2. Connect to the ESXi host console to access the CLI
3. For each vmk and destination device run command:
   • # vmkping -I vmk<x> <destination_IP> -d -s 8972
   • If testing iSCSI destinations this would usually involve two iSCSI destinations
   • If testing vMotion we usually would test all other ESXi hosts in same cluster as current host.
   • Also in all of these commands, for testing a particular stack such as vMotion you can add a string to only test that stack. Example:
     • # vmkping -I vmk<x> -S vmotion <destination_IP> -d -s 8972
4. You can also test if 1500 MTU is working,
   • # vmkping -I vmk<x> <destination_IP> -d -s 1472
5. Additional tests to see if the physical upstream switchport is not configured for jumbo frames: 
   • # vmkping -I vmk<x> <destination_IP> -d -s 1473
   • If the command fails
     • Physical switch value is still at 1500 default value
     • No one made any changes to switch
   • If this command worked but the test for MTU 8972 failed
     • 9000 MTU was set at the switch but didn't allow for correct overhead
     • Overhead differs for different physical switch types
     • You need to check with your network device vendor for correct overhead value

Changing MTU settings

You can do this through the vCenter Server user interface per Enabling Jumbo Frames on virtual switches. To do this through the command line interface (CLI):

1. For each affected standard vSwitch:
   • # esxcfg-vswitch -m <MTU-1500_or_9000> <switchname>
2. For each affected vmk:
   • # esxcfg-vmknic -m <MTU-1500_or_9000> vmk<x>
3. The vmnics will inherit the MTU setting from the switch settings so don't need to be set separately.