"state in doubt; requested fast path state update" error in vmkernel.log
search cancel

"state in doubt; requested fast path state update" error in vmkernel.log

book

Article ID: 318844

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

  • High latency observed on the virtual machines residing on the ESXi host

  • Multiple Hosts enters not responding state, reconnects after 8-9 secs

  • ESXi host UI display error as "Lost access to volume 601####-######-ce7a-############ (DS###_###_LUNS) due to connectivity issues."

  • In the /var/log/vmkernel.log file of the ESXi host, entries similar to below will be seen:

    ####-##-#####:##:##.### <hostname> vmkernel: 116:03:44:19.039 cpu4:4196)<6>qla2xxx 0000:0f:00.0: scsi(6:0:152): Abort command issued -- 1 67a23dcd 2002.
    ####-##-#####:##:##.### <hostname> vmkernel: 116:03:44:19.039 cpu4:4100)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x4100020e0b00) to NMP device "sym.##################" failed on physical path "vmhba2:C0:T0:L152" H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
    ####-##-#####:##:##.### <hostname> vmkernel: 116:03:44:19.039 cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device "sym.##################" state in doubt; requested fast path state update...</time>
    ####-##-#####:##:##.### cpu49:2099091)ScsiDeviceIO: 4115: Cmd(0x45cbdcdf6848) 0x88, CmdSN 0x34d from world 9366736 to dev "naa.600aXXXXXXXXXXXXXXXXXXXXXXXXXXXX" failed H:0x2 D:0x0 P:0x0
    ####-##-#####:##:##.### cpu169:2099095)WARNING: ScsiDeviceIO: 1513: Device naa.600aXXXXXXXXXXXXXXXXXXXXXXXXXXXX performance has deteriorated. I/O latency increased from average value of 1065 microseconds to 22027 microseconds.
    ####-##-#####:##:##.### cpu50:8436000)qlnativefc: vmhba3(33:0.0): qlnativefcStatusEntry:1922:(5:45) Dropped frame(s) detected(524288 of 524288 bytes).

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

These messages appear as the Host Bus Adapter (HBA) driver aborts a command because the command took longer than the timeout period of 5 seconds to complete. An operation can take longer than the timeout period because of several reasons including:
  • Array backup operations (LUN backup, replication, etc.)
  • General overload on the array
  • Read/Write Cache on the array (misconfiguration, lack of cache, etc.)
  • Incorrect tiered storage used (SATA over SCSI)
  • Fabric issues (Bad ISL, outdated firmware, bad fabric cable/GBIC)
Note: The preceding list does not cover the full list of potential reasons as discussing the causes for the Qlogic abort message is not the focus of this Knowledge Base article.
 
 
These message indicates that the command was aborted for scsi(6:0:152), which translates to LUN 152:
 
<YYYY-MM-DD>T<time></time> <hostname> vmkernel: 116:03:44:19.039 cpu4:4196)<6>qla2xxx 0000:0f:00.0: scsi(6:0:152): Abort command issued -- 1 67a23dcd 2002
 
This is our mid-layer noticing that a command aborted, specifically command 0x2a, which is a 10 byte WRITE command. Our mid-layer returns a Host status of 0x2, which translates to DID_BUS_BUSY. This is a catch-all message produced as a direct result of the QLogic driver aborting the command:
 
<YYYY-MM-DD>T<time></time> <hostname> vmkernel: 116:03:44:19.039 cpu4:4100)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x4100020e0b00) to NMP device "sym.##################" failed on physical path "vmhba2:C0:T0:L152" H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
 
When a command is aborted, our mid-layer issues a TUR or TEST_UNIT_READY command down the path in which the command did not complete to ensure that this path is still good to use. A TUR command is issued every 300 seconds down each path as part of the path evaluation code (Disk.PathEvalTime). However, in this instance a TUR command is issued immediately due to the failed command.
 
<YYYY-MM-DD>T<time></time> <hostname> vmkernel: 116:03:44:19.039 cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device "sym.##################" state in doubt; requested fast path state update...
 
In addition to this, when we run the esxtop command on the ESXi host and view the device details by pressing the letter u, in some cases you will see that the device utilization is reaching 100% and the commands are being queued

Resolution

Engage your storage vendor to check if the storage is getting overloaded if it is observed that the device utilization is hitting 100%.
 
If storage utilization is under control and IO aborts are observed, engage your fabric vendor and ensure that the fabric is healthy

Additional Information

Powered by Wolken Software