Windows event alert: The IO Operation at logical block address was retried

Products

• VMware vSphere ESXi

Issue/Introduction

• The following alert is reported in the Windows Event viewer:
  
  The IO operation at logical block address #### for Disk # was retried.
  
  
• When reviewing the /var/run/log/vmkernel.log on the affected ESXi host, warnings, SCSI command aborts (H:0x8 status), and "state in doubt" messages are logged:
  
  [YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu##:######)qlnativefc: vmhba#: qlnativefcEhVirtualReset:###:C#:T#:L#: VIRTUAL RESET ISSUED.
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu##:######)qlnativefc: vmhba#: qlnativefcEhVirtualReset:###:Command aborted on target=###, lun=### - SCSI command timeout counter incremented to ###
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu##:######)qlnativefc: vmhba#: qlnativefcStatusEntry:####:C#:T#:L# - FCP command status: 0x5-0x0 (0x8) portid=#### oxid=0x## cdb=### len=### rspInfo=0x0 resid=0x0 fwResid=0x0 host status = 0x8 device status$
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu##:#####)NMP: nmp_ThrottleLogForDevice:####: Cmd 0x2a (0x##, ##) to dev "<Naa_ID>" on path "vmhba#:C#:T#:L#:" Failed:
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu##:######>)NMP: nmp_ThrottleLogForDevice:####: H:0x8 D:0x0 P:0x0 . Act:EVAL.
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu##:######)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "<Naa_ID>" state in doubt; requested fast path state update...

Environment

VMware  vSphere ESXi 8.x

Cause

The issue occurs because SCSI commands (such as write command 0x2a) sent from the ESXi host to the storage array timeout. This forces the Host Bus Adapter (HBA) driver to abort the I/O requests, as indicated by host status code H:0x8.

Resolution

Engage the storage vendor to check if the storage is getting overloaded or the device utilization is hitting 100%.

If storage utilization is under control and IO aborts are observed, engage your fabric vendor and ensure that the fabric is healthy.