iSCSI storage latency on ESXi hosts is reported very high

Products

• VMware vSphere ESXi

Issue/Introduction

• Poor storage performance is observed on VMs and ESXi hosts

• Storage connectivity is via software iSCSI

• Analyzing the esxtop metric reveals that the device latency is higher than 30 ms. To analyze "esxtop" metrics follow the below steps:

• • SSH to the ESXi host and launch esxtop by running the command 'esxtop' and press "u" to view storage device statistics. Confirm the following latency thresholds:
     • DAVG (Device Average Latency) > 30ms
     • KAVG (Kernel Average Latency) < 2ms
    
    

• Using the following command to check for packet loss, it is observed that ping replies from storage array IP have a round trip time (rtt) of less than 1 ms:
  vmkping -I <vmk#> -s <MTU> -d <iscsi_target_IP>

• The physical network adapter's driver and firmware are compatible.
  Refer to the VMware article: “Determining Network/Storage firmware and driver version in ESXi” for validation steps. 
• "High QLEN (Queue Length) observed in esxtop storage view" and "Intermittent 'Lost access to volume' due to datastore heartbeat timeouts."

Environment

VMware vSphere ESXi

Cause

This is caused by a physical switch setting known as storm control that protects against congestion.

Resolution

Please work with the hardware vendor associated with your physical network switch, to reconfigure the storm control settings on the VLAN used for iSCSI, appropriately to allow iSCSI traffic to traverse the switch without being throttled.

Please consider to disable iSCSI Port Binding when adapters are on different network segments and to set Network Failback to "No" to mitigate effects of flapping physical ports.