Users experiencing poor storage performance despite minimal load.

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Users report poor storage performance despite minimal load on the devices.

• Storage connectivity is via software iSCSI.

• Device latency is low/as expected (as observed via vCenter performance charts or esxtop).  

Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

Cause

While latency is low on successful I/O operations, transient storage conditions erors are causing performance degradation. In repsonse to transient storage conditions, means I/O requeuing due to temporary storage unavailability or delays.

Verification:

/var/log/vmkernel.log indicates repeated transient storage condition issues via H:0xc SCSI code:

YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu4:2097302)ScsiDeviceIO: 4580: Cmd(0x45bb406a9f80) 0x8a, CmdSN 0x3ad from world 93812680 to dev "naa.#################" failed H:0xc D:0x0 P:0x0 
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu20:2097318)ScsiDeviceIO: 4580: Cmd(0x45bb4073a000) 0x8a, CmdSN 0x312 from world 93812680 to dev "naa.#################" failed H:0xc D:0x0 P:0x0
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu82:2097374)ScsiDeviceIO: 4580: Cmd(0x45bafc42ee00) 0x8a, CmdSN 0x3de from world 93812680 to dev "naa.#################" failed H:0xc D:0x0 P:0x0
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu82:2097374)ScsiDeviceIO: 4580: Cmd(0x45bafc490280) 0x8a, CmdSN 0x3bc from world 93812680 to dev "naa.#################" failed H:0xc D:0x0 P:0x0

 

 

Resolution

To address the transient storage condition and improve overall storage performance, perform the following steps:

• Review storage array logs and health status to check for intermittent availability or performance degradation.
• Work with the storage vendor to identify any underlying issues such as firmware bugs, controller failovers, or congestion.
• Verify that the physical network adapter's driver and firmware are compatible.
  Refer to the VMware article: “Determining Network/Storage firmware and driver version in ESXi” for validation steps.  
• Ensure there are no intermittent connectivity issues between the ESXi host and the storage array. Troubleshooting ESXi  connectivity to iSCSI arrays using software initiators 
• Check for interface flapping, CRC errors, or packet drops on physical switches and network adapters.