I/O to iSCSI storage continuously fails with 'Cancelled from driver layer' error

Products

• VMware vSphere ESXi

Issue/Introduction

• In /var/run/log/vmkernel.log, there are continuous failures of SCSI commands with 'Cancelled from driver layer':
  
  YYYY-mm-ddTHH:MM:SS.XXXZ cpu#:#######)NMP: nmp_ThrottleLogForDevice:3867: Cmd 0x12 (0x45b921633a08, 0) to dev "naa.################################" on path "vmhba##:C#:T#:L#" Failed:
YYYY-mm-ddTHH:MM:SS.XXXZ cpu#:#######)NMP: nmp_ThrottleLogForDevice:3875: H:0x5 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x430768c2e2d0 CmdSN 0x######
YYYY-mm-ddTHH:MM:SS.XXXZ cpu#:#######)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.################################" state in doubt; requested fast path state update...
YYYY-mm-ddTHH:MM:SS.XXXZ cpu#:#######)ScsiDeviceIO: 4154: Cmd(0x45b921633a08) 0x12, cmdId.initiator=0x430768c2e2d0 CmdSN 0x###### from world # to dev "naa.################################" failed H:0x5 D:0x0 P:0x0 Cancelled from driver layer. Cmd
YYYY-mm-ddTHH:MM:SS.XXXZ cpu#:#######)count Active:1 Queued:0
  
  
  
• Storage is connected via iSCSI.
  
  
  
• No driver (iscsi_vmk) error seen in vmkernel.log
  
  
  
• Packet capture shows that the target storage continuously reports MaxCmdSN < ExpCmdSN
  
  iSCSI (NOP In) 
    ..10 0000 = Opcode: NOP In (0x20)
    ::
    ExpCmdSN: 570302113
    MaxCmdSN: 570302112
  
  

Cause

MaxCmdSN < ExpCmdSN means the ESXi host has already sent commands up to MaxCmdSN, and the window size is 0. Consequently, ESXi cannot send additional I/O to the targets.

Resolution

Engage the storage vendor support to investigate why the target does not increment MaxCmdSN.