Slowness/latency reported on multiple VMs.

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

Latency reported by multiple end users.
VMkernel logs confirm I/O latency:
YYYY-MM-DDTHH:MM.SSSZ Wa(180) vmkwarning: cpu65:2098718)WARNING: ScsiDeviceIO: 1779: Device naa.################################ performance has deteriorated. I/O latency increased from average value of 1545 microseconds to 36483 microseconds.
YYYY-MM-DDTHH:MM.SSSZ Wa(180) vmkwarning: cpu1:2098717)WARNING: ScsiDeviceIO: 1779: Device naa.################################performance has deteriorated. I/O latency increased from average value of 3009 microseconds to 84897 microseconds.
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu32:2098717)ScsiDeviceIO: 1779: Device naa.################################performance has improved. I/O latency reduced from 84897 microseconds to 12077 microseconds.
YYYY-MM-DDTHH:MM.SSSZ Wa(180) vmkwarning: cpu1:2098719)WARNING: ScsiDeviceIO: 1779: Device naa.################################performance has deteriorated. I/O latency increased from average value of 3009 microseconds to 65020 microseconds.

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

LUN unable to accept the SCSI commands from initiators due to lack of resources which resulted in I/O failures.

Cause validation:

var/run/log/Vmkernel.log:

I/O failures reported with SCSI sense code D:0x28, which indicates that the LUN is unable to accept SCSI commands from initiators due to lack of resources, namely the queue depth on the array.
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu1:2098726)NMP: nmp_ThrottleLogForDevice:3893: Cmd 0x8a (0x45bb4b55ce00, 14988033) to dev "naa.################################" on path "vmhba4:C0:T3:L2" Failed:
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu1:2098726)ScsiDeviceIO: 4619: Cmd(0x45bb4b55ce00) 0x8a, CmdSN 0x80000076 from world 14988033 to dev "naa.################################" failed H:0x0 D:0x28 P:0x0
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu44:2098726)ScsiDeviceIO: 4619: Cmd(0x45bb5be955c0) 0xfe, CmdSN 0x1898609 from world 2097288 to dev "naa.################################" failed H:0x0 D:0x28 P:0x0
Read (0x28) and write (0x8a) I/O failures observed with scsi codes "H:0x5 and H0x8". H:0x5 indicates that the driver has to abort commands in-flight to the target. This can occur due to a command timeout or parity error in the frame. H:0x8 will be reported when the HBA driver has aborted the I/O. It can also occur if the HBA does a reset of the target.

YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu49:2098726)ScsiDeviceIO: 4656: Cmd(0x45bb5414bf40) 0xfe, cmdId.initiator=0x430bba9ee080 CmdSN 0x18ebfef from world 2097288 to dev "naa.################################" failed H:0x5 D:0x0 P:0x0 . Cmd count Active:19
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu89:15375775)NMP: nmp_ThrottleLogForDevice:3893: Cmd 0x28 (0x45bb327afb40, 2099647) to dev "naa.################################" on path "vmhba4:C0:T3:L2" Failed:
YYYY-MM-DDTHH:MM.SSSZ Wa(180) vmkwarning: cpu89:15375775)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "naa.################################" state in doubt; requested fast path state update...
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu89:15375775)ScsiDeviceIO: 4591: Cmd(0x45bb327afb40) 0x28, cmdId.initiator=0x430bba9ee080 CmdSN 0x18ec002 from world 2099647 to dev "naa.################################" failed H:0x8 D:0x0 P:0x0 Cancelled from path layer
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu89:15375775)ScsiDeviceIO: 4591: Cmd(0x45bb37a67400) 0x8a, cmdId.initiator=0x43108b9a2ac0 CmdSN 0x80000068 from world 14988033 to dev "naa.################################" failed H:0x8 D:0x0 P:0x0 Cancelled from path layer
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu89:15375775)ScsiDeviceIO: 4591: Cmd(0x45bb53531c00) 0x8a, cmdId.initiator=0x43108b9a2ac0 CmdSN 0x80000025 from world 14988033 to dev "naa.################################" failed H:0x8 D:0x0 P:0x0 Cancelled from path layer
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu89:15375775)ScsiDeviceIO: 4591: Cmd(0x45bb37a3c200) 0x8a, cmdId.initiator=0x43108b9a2ac0 CmdSN 0x8000007c from world 14988033 to dev "naa.################################" failed H:0x8 D:0x0 P:0x0 Cancelled from path layer
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu89:15375775)ScsiDeviceIO: 4591: Cmd(0x45bb54192740) 0x8a, cmdId.initiator=0x43108b9a2ac0 CmdSN 0x80000075 from world 14988033 to dev "naa.################################" failed H:0x8 D:0x0 P:0x0 Cancelled from path layer
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu89:15375775)ScsiDeviceIO: 4591: Cmd(0x45bb32739f40) 0x8a, cmdId.initiator=0x43108b9a2ac0 CmdSN 0x80000044 from world 14988033 to dev "naa.################################" failed H:0x8 D:0x0 P:0x0 Cancelled from path layer
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu18:2098724)ScsiDeviceIO: 4591: Cmd(0x45bb4b58fe00) 0x8a, cmdId.initiator=0x430bba9ee080 CmdSN 0x18ebff7 from world 15508927 to dev "naa.################################" failed H:0x8 D:0x0 P:0x0 Cancelled from driver
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu18:2098724)ScsiDeviceIO: 4591: Cmd(0x45bb32731540) 0x8a, cmdId.initiator=0x430bba9ee080 CmdSN 0x18ebffb from world 15508928 to dev "naa.################################" failed H:0x8 D:0x0 P:0x0 Cancelled from driver

Validate FC stats by running below cmd

# esxcli storage san fc stats get 
FcStat:
   Adapter: vmhba#
   Tx Frames: 19687
   Rx Frames: 693458
   Lip Count: 0
   Error Frames: 0
   Dumped Frames: 0
   Link Failure Count: 793
   Loss of Signal Count: 0
   PrimSeq Protocol Err Count: 0
   Invalid Tx Word Count: 186
   Invalid CRC Count: 0
   Input Requests: 0
   Output Requests: 0
   Control Requests: 0

Invalid CRC count may also be high.

Resolution

Involve storage vendor to further investigate.

Additional Information

Validate the driver/Firmware compatibility using Broadcom Compatibility Guide