High disk latency observed on Dell MD3000i storage array

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• You observe very high disk latency on one or more LUNs when viewing disk statistics on esxtop.
• This high disk latency can often present as performance problems for one or more virtual machines.
• The high latency does not persist and the latency drops back to acceptable disk latency (DAVG/cmd value of 10ms or less).
• High disk latency occurs when doing a high I/O task such as storage vMotion, cloning a disk or virtual machine, or a large snapshot consolidation.

Environment

VMware ESXi 4.1.x Installable
VMware ESXi 4.1.x Embedded
VMware ESX 4.0.x
VMware ESXi 4.0.x Installable
VMware ESX 4.1.x
VMware ESXi 4.0.x Embedded

Cause

This issue can occur if Flow Control is not enabled on the physical switch.

For more information, see the Dell document IP SAN Best Practices.
 

Resolution

Confirm that your iSCSI configuration as per Dell best practice for the MD3000i storage array. For more information, see the Dell Wiki article http://en.community.dell.com/techcenter/virtualization/w/wiki/3043.aspx.

To verify that Flow Control is enabled on your physical network switches:

1. Check each host in your environment. Click Configuration > Networking to determine each iSCSI port group that you have configured.
2. For each vmnic network card connected to these port groups, trace the network cables back to the port on the physical network switch.
   
   You may also be able to identify the port either by using CDP (blue icon next to the network cards in vSphere client) if CDP is enabled.
   
   Your physical switch may show the MAC/Ethernet addresses of the network cards connected to each port. The Network Adapters section on the vSphere client shows the MAC address of each vmnic.
    
3. For each port on the physical network switch that the vmnic is connected to, ensure that Flow Control is enabled.

If Flow Control is enabled on all ports on the physical switch and the problem of high latency remains, open a case with both Dell and VMware Support to continue troubleshooting this issue.