Understanding SCSI Target NMP Errors and Conditions in ESXi

Products

VMware vSphere ESXi

Issue/Introduction

This article provides information about the SCSI device status messages that you may observe when reviewing NMP errors in the VMkernel log.

Symptoms:

Due to these status ESXi hosts can go in unresponsive state
ESXi Hosts can face Datastore disconnections
VMs might get in to the unresponsive / invalid or orphan state

Environment

VMware vSphere ESXi 8.x

VMware vSphere ESXi 7.x

VMware vSphere ESXi 6.x

Resolution

These are SCSI device/target status messages that are defined, updated, and maintained by t10.org.

SCSI device/target status

`VMK_SCSI_DEVICE_GOOD = 0x0`

vmkernel: 116:03:44:19.039 cpu4:4100)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x4100020e0b00) to NMP device "sym.029010111831353837" failed on physical path "vmhba2:C0:T0:L152" H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

This status is returned when there is no error from the device or target side. This is when you will see if there is a status for Host or Plugin.

For more information on host side NMP errors, see Understanding SCSI host-side NMP errors/conditions in VMware ESXi

`VMK_SCSI_DEVICE_CHECK_CONDITION = 0x2`

vmkernel: 76:23:45:36.239 cpu11:22687)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x41000b10f000) to NMP device "naa.600601601f70190016361881f3b4de11" failed on physical path "vmhba3:C0:T2:L10" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x4 0x3.

This status is returned when a command fails for a specific reason. When a CHECK CONDITION is received, the ESX storage stack will send out a SCSI command 0x3 (REQUEST SENSE) in order to get the SCSI sense data (Sense Key, Additional Sense Code, ASC Qualifier, and other bits). The sense data is listed after Valid sense data in the order of Sense Key, Additional Sense Code, and ASC Qualifier. When decoding Additional Sense Code/ASC Qualifier bits, they must be read together. Here is a breakdown of the above message:

Host Status = 0x0 = OK
Device Status = 0x2 = Check Condition
Plugin Status = 0x0 = OK

Sense Key = 0x2 = NOT READY
Additional Sense Code/ASC Qualifier = 0x4/0x3 = LOGICAL UNIT NOT READY - MANUAL INTERVENTION REQUIRED

For a complete list of possible Sense Keys, see SCSI Sense Keys
For a complete list of Additional Sense Code/ASC Qualifier pairings, see ASC-NUM.TXT

`VMK_SCSI_DEVICE_CONDITION_MET = 0x4`

vmkernel: 0:00:42:28.062 cpu5:4112)ScsiDeviceIO: SCSIAsyncDeviceCommand:2077: Error Inject: Command 0x2a to device "naa.600601600f6021002cfb75ea2bc6de11" failed H:0x0 D:0x4 P:0x0

This status is returned for successful completion of conditional commands, such as PREFETCH.

`VMK_SCSI_DEVICE_BUSY = 0x8`

vmkernel: 1:02:02:02.206 cpu3:4099)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x410005078e00) to NMP device "naa.6001e4f000105e6b00001f14499bfead" failed on physical path "vmhba1:C0:T0:L100" H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.

This status is returned when a LUN cannot accept SCSI commands at the moment. As this should be a temporary condition, the command is tried again.

`VMK_SCSI_DEVICE_INTERMEDIATE = 0x10`

Never seen in ESX.

This status is the same as 0x0 (GOOD) and is returned for a command that was part of a series of linked commands.

`VMK_SCSI_DEVICE_INTERMEDIATE_CONDITION_MET = 0x14`

Never seen in ESX.

This status is the combination of device statuses CONDITION MET (0x4) and INTERMEDIATE (0x10).

`VMK_SCSI_DEVICE_RESERVATION_CONFLICT = 0x18`

vmkernel: 1:08:40:03.933 cpu5:4736)ScsiDeviceToken: 115: Completed IO with status H:0x0 D:0x18 P:0x0 after losingreservation on device naa.6006016026601d007c174a7aa292df11

This status is returned when a LUN is in a Reserved status and commands from initiators that did not place that SCSI reservation attempt to issue commands to it. Rarely will you ever see a device status of 0x18 in the format shown above. The most common form of a RESERVATION CONFLICT status message is seen as follows:

For more information on troubleshooting SCSI Reservation conflicts, see Analyzing SCSI Reservation conflicts on VMware ESXi

`VMK_SCSI_DEVICE_COMMAND_TERMINATED = 0x22`

Never seen in ESX.

Obsolete status code. Was originally returned as a result of a TERMINATE I/O message. That message is also obsolete.

`VMK_SCSI_DEVICE_QUEUE_FULL (TASK SET FULL) = 0x28`

vmkernel: 1:08:42:28.062 cpu3:8374)NMP: nmp_CompleteCommandForPath:2190: Command 0x16 (0x41047faed080) to NMP device "naa.600508b40006c1700001200000080000" failed on physical path "vmhba39:C0:T1:L16" H:0x0 D:0x28 P:0x0 Possible sense data: 0x0 0x0 0x0.

This status is returned when the LUN prevents accepting SCSI commands from initiators due to lack of resources, namely the queue depth on the array.

Adaptive queue depth code was introduced into ESX 3.5 U4 (native in ESX 4.x) that adjusts the LUN queue depth in the VMkernel. If configured, this code will activate when device status TASK SET FULL (0x28) is return for failed commands and essentially throttles back the I/O until the array stops returning this status.

For more information, see Controlling LUN queue depth throttling in VMware ESXi.

`VMK_SCSI_DEVICE_ACA_ACTIVE = 0x30`

vmkernel: 0:00:14:08.673 cpu2:4098)NMP: nmp_CompleteCommandForPath: Command 0x25 (0x41013f5b4a40) to NMP device "naa.60060160f1c01400d2731d49473edd11" failed on physical path "vmhba2:C0:T1:L0" H:0x0 D:0x30 P:0x0 Possible sense data: 0x0 0x0 0x0.

This status is returned when an Auto Contingent Allegiance (ACA) has been aborted by another SCSI initiator. This status has been observed on arrays running older/unsupported firmware with ESX 4.1, specifically because the firmware does not support VAAI commands.

For information on VAAI, see vStorage APIs for Array Integration FAQ .

`VMK_SCSI_DEVICE_TASK_ABORTED = 0x40`

vmkernel: 1:01:32:19.519 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x4100062e7d00) to NMP device "naa.60a9800043346534645a43396a6c524d" failed on physical path "vmhba3:C0:T0:L23" H:0x0 D:0x40 P:0x0 Possible sense data: 0x0 0x0 0x0.

This status is returned when a command has been aborted by another SCSI initiator and the TASK ABORT STATUS bit is set to '1'. So far this status has only been returned by iSCSI arrays.

Any device status value not covered in this list would be vendor specific. If a vendor specific condition is hit, please contact the vendor support team for clarification on the status.

Op-Codes :

opcode 28

vmkernel: 12:17:46:05.000 cpu12:4108)ScsiCore: 1181: Sync CR (opcode 28) at 992 (wid 4121)

The opcode or SCSI Command Operation Code referenced in this message is 28 or 0x28. This is a READ(10) command or a 10 byte READ, which is a command to be issued by a virtual machine.

For more information on SCSI Command Operation Codes, see OP-NUM