Pure Storage array returns Check Condition, Aborted Command when receiving an out of order Fiber Channel Protocol (FCP) frame

Products

• VMware vSphere ESX 6.x
• VMware vSphere ESX 7.x
• VMware vSphere ESX 8.x
• VMware vSphere ESXi
• VMware vSphere ESXi 8.0

Issue/Introduction

A VCF Administrator will observe SCSI command failures in the /var/log/vmkernel.log for LUNs from the PURE storage array:

NMP: nmp_ThrottleLogForDevice:3867: Cmd 0x8a (0x45dab79b9a08, 3367783) to dev "naa.624a93701b68843#################" on path "vmhba1:C0:T1278:L248" Failed:
NMP: nmp_ThrottleLogForDevice:3875: H:0x0 D:0x2 P:0x0 Valid sense data: 0xb 0x29 0x3. Act:NONE. cmdId.initiator=0x430efb03e780 CmdSN 0x80000052
ScsiDeviceIO: 4115: Cmd(0x45dab79b9a08) 0x8a, CmdSN 0x80000052 from world 3367783 to dev "naa.624a93701b68843#################" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xb 0x29 0x3
NMP: nmp_ThrottleLogForDevice:3867: Cmd 0x8a (0x45baa137cb08, 2104257) to dev "naa.624a93701b68843#################" on path "vmhba1:C0:T1279:L220" Failed:
NMP: nmp_ThrottleLogForDevice:3875: H:0x0 D:0x2 P:0x0 Valid sense data: 0xb 0x29 0x3. Act:NONE. cmdId.initiator=0x430efb024080 CmdSN 0x80000062
ScsiDeviceIO: 4115: Cmd(0x45baa137cb08) 0x8a, CmdSN 0x80000062 from world 2104257 to dev "naa.624a93701b68843#################" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xb 0x29 0x3

Environment

ESXi (Any Version)

PURE Storage array

Cause

The Sense Key (0xb) and Additional Sense Code/ASC Qualifier (0x29/0x3) translates to:
---------------------------
0b: ABORTED COMMAND
29/03: BUS DEVICE RESET FUNCTION OCCURRED

Pure Storage exclusively uses a Sense Key of 0xb and an ASC/ASCQ of 0x29/0x3 for when it receives an out of order FCP frame, which translates to a dropped frame. In this way, the Pure Storage array is able to notify that the reason for the command failure on its end was effectively due to a dropped frame. No other storage array vendors uses a Sense Key of 0xb with a ASC/ASCQ of 0x29/0x3 to highlight that a command has failed due to a dropped frame.

Resolution

Contact Pure Storage support for assistance in determining the cause of these dropped frames. Dropped frames are going to be a physical layer issue (bad SFP, bad cable, low light levels, etc) so the Fabric Switches should be reviewed as well for CRC or transmit errors.