Troubleshooting disk latency when using Jumbo Frames with iSCSI or NFS datastores

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides information on how to isolate the issue when igh disk latency is observed when using Jumbo Frames with iSCSI or NFS datastores.

Environment

VMware vSphere ESXi

Cause

The disk latency may happen if the storage processor or ESX/ESXi host is not configured properly for the MTU size which was selected.

Resolution

To ensure the host is configured properly for the defined MTU size:

1. Log in to ESX or ESXi host using SSH or Tech Support Mode. See: Using ESXi Shell in ESXi (311213)
   
   
2. Run this command from the ESX/ESXi host:
   
   vmkping -I <vmkernel_used_for_iSCSI> IP_address_of_NFS_or_iSCSI_server -s MTU_header_size -d 
   
   Where:
   -s sets the packet size
   -d indicates do not fragment the packet
   
   Note: Assuming header size = 216, then -s value (packet size) would be: 9000 - 216 = 8784 and interface vmk4 is used for iSCSI
   Example command:
   
   # vmkping -I vmk4 ###.###.##.# -s 8784 -d  
   
   
3. If a response is received, this means that the communication is occurring at a desired MTU. If no response is received, run the vmkping command without -d option:
   
   # vmkping -I <vmkernel_used_for_iSCSI> IP_address_of_NFS_or_iSCSI_server -s MTU_header_size
   
   
4. If there is a response, this means that the configuration issue still exists, but that the large packets are being fragmented. This issue may lead to disk latency, the disruption of networking or storage for other components in the environment. Fragmenting and re-assembling packets uses a lot of CPU resources on the switch, storage processor, and ESX hosts.
   
   
5. Verify that the ESX/ESXi host is configured properly for Jumbo Frames. Run the below command:
   
   # esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:##:##.00 bnx2 Up 1000Mbps Full xx:xx:xx:xx:xx:xx 1500 Broadcom Corporation PowerEdge R710 BCM5709 Gigabit Ethernet
vmnic1 0000:##:##.01 bnx2 Up 1000Mbps Full xx:xx:xx:xx:xx:xy 1500 Broadcom Corporation PowerEdge R710 BCM5709 Gigabit Ethernet
vmnic2 0000:##:##.02 bnx2 Up 1000Mbps Full xx:xx:xx:xx:xx:xz 9000 Broadcom Corporation PowerEdge R710 BCM5709 Gigabit Ethernet
   
   
6. Verify that the vSwitch is configured for Jumbo Frames once a value is found under the MTU column that matches the desired MTU size.
   
   # esxcfg-vswitch -l
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 128 3 128 1500 vmnic0

PortGroup Name VLAN ID Used Ports Uplinks
Management Network 0 1 vmnic0

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch1 128 2 128 1500 vmnic1

PortGroup Name VLAN ID Used Ports Uplinks
VMnet VLAN 5 0 0 vmnic1

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch2 128 3 128 9000 vmnic2

PortGroup Name VLAN ID Used Ports Uplinks
iSCSI_1 0 1 vmnic2
   
7. Verify the MTU column for the vSwitch that has the VMkernel port configured on it also matches the MTU size. For more information, see iSCSI and Jumbo Frames configuration on ESX/ESXi (1007654) or Enabling Jumbo Frames for VMkernel ports in a virtual distributed switch (1038827).
   
   Note: If the VMkernel or vSwitch are configured for Jumbo Frames, then there is a configuration problem either on a network component such as network switches or routers, or with the storage processor.
   
   
8. Verify that all devices between the ESX host and the storage array (including physical network switches) are configured to support Jumbo Frames for the desired MTU size.

Additional Information

For more information, please refer article: Troubleshooting ESXi connectivity to iSCSI arrays using software initiators