iSCSI and Jumbo Frames configuration on VMware ESXi/ESX

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides information on setting up a NAS and an iSCSI environment with Jumbo Frames.

Symptoms:

• You see large queues when monitoring Jumbo Frames.
• You observe packet fragmentation when viewing a packet capture using the tcpdump utility.

Environment

VMware vSphere 7.0.x
VMware vSphere 8.0.x

Resolution

Ensure that you read this important information about Jumbo Frames before working with them:

• ESX/ESXi supports a maximum MTU size of 9000.
  
  Note: Some switch configurations for Jumbo Frames need to have an MTU set higher than 9000. For more information, see the Cisco Nexus 5000 Series NX-OS Software Configuration Guide.
  
  Note: The preceding link is correct as of April 14, 2015. If you find the link is broken, provide a feedback and a VMware employee will update the link.
  
• Any packet larger than 1500 MTU is a Jumbo Frame. ESX/ESXi supports frames up to 9000 Bytes.
• It is possible to configure Jumbo Frames for an iSCSI network. It is not a supported configuration in ESXi/ESX 3.5, but it is supported in ESXi/ESX 4.x, ESXi 5.x and 6.0.
• You can enable Jumbo Frames for each vSwitch or VMkernel interface through the command line interface on your ESX host.
• To allow an ESX host to send larger frames out onto the physical network, the network must support Jumbo Frames end to end.
• Ensure that your NIC or LOM supports Jumbo Frames.
• For experimental support of Jumbo Frames in ESX 3.5, these NICs are supported:
  
  
  • Intel (82546, 82571)
  • Broadcom (5708, 5706, 5709, 57710, 57711)
  • Netxen (NXB-10GXxR, NXB-10GCX4)
  • Neterion (Xframe, Xframe II, Xframe E)
    
    
• For ESXi/ESX 4.x, ESXi 5.x and 6.0, contact your NIC hardware vendor regarding support for Jumbo Frames.

Jumbo Frames in ESXi 5.1 and later

Jumbo frames for all iSCSI adapters in vSphere 5.1, vSphere 5.5 and vSphere 6.0 can be configured using the user interface (UI). This applies to Software iSCSI, Dependent Hardware iSCSI and Independent Hardware iSCSI adapters.

 

To enable Jumbo Frames for software and dependent hardware iSCSI adapters in the vSphere Web Client, change the default value of the MTU parameter:

1. Browse to the host in the vSphere Web Client navigator.
2. Click the Manage tab.
3. Click Networking.
4. Click Virtual Switches, and select the vSphere switch that you want to modify from the list.
5. Click Edit Settings.
6. On the Properties page, change the MTU parameter.
   
   Note: This step sets the MTU for all physical NICs on that standard switch. The MTU value should be set to the largest MTU size among all NICs connected to the standard switch.

 

To enable Jumbo Frames for independent hardware iSCSI adapters in the vSphere Web Client, change the default value of the MTU parameter:

 

Use the Advanced Options settings to change the MTU parameter for the iSCSI HBA.

1. Browse to the host in the vSphere Web Client navigator.
2. Click the Manage tab.
3. Click Storage.
4. Click Storage Adapters, and select the independent hardware iSCSI adapter from the list of adapters.
5. Under Adapter Details, click the Advanced Options tab.
6. Click Edit.
7. Change the value of the MTU parameter.

 

Configuring Jumbo Frames on a vSphere Standard Switch

To configure Jumbo Frames on a vSphere Standard Switch:

1. Log in to the vSphere Client.
2. Select the Hosts and Clusters inventory view.
3. Under the host Configuration tab, click Networking.
4. Click Properties for the vSphere standard switch associated with the VMkernel to modify.
5. Under the Ports tab, select the VMkernel interface.
6. Click Edit.
7. Set the MTU parameter to 9000.
8. Click OK.

Note: To create a Jumbo Frames-enabled vNetwork Distributed Switch and its associated VMkernel interfaces, see Enabling Jumbo Frames for VMkernel ports in a virtual distributed switch (324494)

To create a Jumbo Frames-enabled VMkernel interface on a vNetwork Standard Switch:

1. Log directly in to the ESX host console.
2. Obtain the current vSwitch and portgroup configuration with the esxcfg-vswitch command:
   
   # esxcfg-vswitch -l
   
   
3. To create a VMkernel interface with Jumbo Frames support, you must first create a portgroup on an existing vSwitch:
   
   # esxcfg-vswitch -A vmkernel_port_group_namevSwitch#
   
   Example:
   
   # esxcfg-vswitch -A iSCSI vSwitch2
   
   Note: If you plan to have a vSwitch that contains only the iSCSI port group, you can specify the MTU for the vSwitch to be 9000 and must specify an MTU of 9000 when creating the VMkernel port as well in the next step. To configure a vSwitch to use Jumbo Frames (MTU 9000):
   
   # esxcfg-vswitch -m 9000 vSwitch#
   
   Example:
   
   # esxcfg-vswitch -m 9000 vSwitch2
   
   
4. To create a VMkernel connection with Jumbo Frame support:
   
   
   • Run this command for ESX 3.5 and ESXi/ESX 4.x:
     
     # esxcfg-vmknic -a -i ip_address -n netmask -m MTU portgroup_name
     
     Example:
     
     # esxcfg-vmknic -a -i 192.168.#.# -n 255.#.#.# -m 9000 iSCSI
     
     Note: If the vmnic port is already created, use the command for ESXi/ESX 4.1 only:
     
     # esxcfg-vmknic -m 9000 portgroup_name
     
     Example:
     
     # esxcfg-vmknic -m 9000 iSCSI
     
     
   • Run this command for ESXi 5.x and 6.0:
     
     # esxcli network ip interface set -m 9000 -i vmk_interface
     
     Example:
     
     # esxcli network ip interface set -m 9000 -i vmk1
     
     
5. To display a list of VMkernel interfaces, and to check that the configuration of the Jumbo Frame‐enabled interface is correct:
   
   
   • Run this command for ESX 3.5 and ESXi/ESX 4.x:
     
     # esxcfg-vmknic -l
     
     
   • Run this command for ESXi 5.0 and 6.0:
     
     # esxcli network ip interface list
     
     

Additional Information

For related information, see Enabling and verifying IOAT and Jumbo frames (344164).


Enabling Jumbo Frames on virtual distributed switches
Configuração de iSCSI e Jumbo Frames em ESX/ESXi
在 VMware ESXi/ESX 上配置 iSCSI 帧和巨帧