Repeated All-Paths-Down (APD) errors on Micron NVME drives
search cancel

Repeated All-Paths-Down (APD) errors on Micron NVME drives

book

Article ID: 412411

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware vSphere ESXi 8.0

Issue/Introduction

  • The ESXi host is configured with locally installed NVMe drives.
  • The hosts are reporting "All-Paths-Down" status for the NVMe drive(s):

[YYYY-MM-DDTHH:MM:SS]: In(14) vobd[2099683]:  [vmfsCorrelator] 6927271523835us: [vob.vmfs.heartbeat.timedout] ########-########-####-############ <DATASTORE_NAME>
[YYYY-MM-DDTHH:MM:SS]: In(14) vobd[2099683]:  [vmfsCorrelator] 6927390774021us: [esx.problem.vmfs.heartbeat.timedout] ########-########-####-############ <DATASTORE_NAME>
[YYYY-MM-DDTHH:MM:SS]: In(14) vobd[2099683]:  [APDCorrelator] 6927311909286us: [vob.storage.apd.start] Device or filesystem with identifier [t10.NVMe____#########################________________#########################] has entered the All Paths Down state.
[YYYY-MM-DDTHH:MM:SS]: In(14) vobd[2099683]:  [psastorCorrelator] 6927311909218us: [vob.psastor.psastorpath.pathstate.dead] storagePath vmhba1:C0:T0:L0 changed state from on (device ID: t10.NVMe____#########################________________#########################)
[YYYY-MM-DDTHH:MM:SS]: In(14) vobd[2099683]:  [APDCorrelator] 6927431159954us: [esx.problem.storage.apd.start] Device or filesystem with identifier [t10.NVMe____#########################________________#########################] has entered the All Paths Down state.
[YYYY-MM-DDTHH:MM:SS]: In(14) vobd[2099683]:  [psastorCorrelator] 6927431168453us: [esx.problem.storage.connectivity.lost] Lost connectivity to storage device t10.NVMe____#########################________________#########################. Path vmhba1:C0:T0:L0 is down. Affected datastores: "<DATASTORE_NAME>".
[YYYY-MM-DDTHH:MM:SS]: In(14) vobd[2099683]:  [APDCorrelator] 6927451910165us: [vob.storage.apd.timeout] Device or filesystem with identifier [t10.NVMe____#########################________________#########################] has entered the All Paths Down Timeout state after being in the All Paths Down state for 140 seconds. I/Os will now be fast failed.
[YYYY-MM-DDTHH:MM:SS]: In(14) vobd[2099683]:  [APDCorrelator] 6927571163551us: [esx.problem.storage.apd.timeout] Device or filesystem with identifier [t10.NVMe____#########################________________#########################] has entered the All Paths Down Timeout state after being in the All Paths Down state for 140 seconds. I/Os will now be fast failed.

  • Entries similar to the following appear in the vmkernel.log file, displaying a 0x808 status code:

[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu###:2100026)WARNING: HPP: HppNvmeThrottleLogForDevice:600: NVMe Cmd 0x2 (0x45db81be0cc0, 4358304) to dev "t10.NVMe____#########################________________#########################" on path "vmhba1:C0:T0:L0" Failed:
[YYYY-MM-DDTHH:MM:SS]: Wa(180) vmkwarning: cpu###:2100026)WARNING: HPP: HppNvmeThrottleLogForDevice:608: Error status H:0x9 D:0x0 P:0x0 hppAction = 1
[YYYY-MM-DDTHH:MM:SS]: Wa(180) vmkwarning: cpu###:2100026)WARNING: NVMEPSA:217 Complete vmkNvmeCmd: 0x45bb8f7a90c0, vmkPsaCmd: 0x45db81be0cc0, cmdId.initiator=0x4311c8c17d80, CmdSN: 0x120, status: 0x808

  • Additionally, the logs may record "Virtual resets" being issued to the NVMe device alongside aborted command tasks:

[YYYY-MM-DDTHH:MM:SS]: In(182) vmkernel: cpu3:2100134)NvmeDeviceIO: 3315: Virt reset issued on device t10.NVMe____#########################________________#########################
[YYYY-MM-DDTHH:MM:SS]: In(182) vmkernel: cpu103:2099594)NVMEPSA:1345 taskMgmt:abort cmdId.initiator=0x430c72ac9f00 CmdSN 0x557fe25 world:2097544 controller 261 state:5 nsid:1
[YYYY-MM-DDTHH:MM:SS]: In(182) vmkernel: cpu103:2099594)NVMEIO:3974 Ctlr 261, ns 1, tmReq 0x431f44f2c320, type 1, initiator 0x430c72ac9f00, sn 0x557fe25, world id 2097544.
[YYYY-MM-DDTHH:MM:SS]: In(182) vmkernel: cpu103:2099594)NvmeUtil: 470: Transient status for command 0x5 set to VMK_ABORTED because retries were inhibited (e.g., by a guest virtual device reset): cmdId.initiator=0x430c72ac9f00 cmdId.serialNumber=0x557fe25)

  • The I/O being issued to the datastore is minimal.

Environment

VMware vSphere ESXi 8.0.x

Cause

  • The "status: 0x808" error indicates that the NVMe drives are currently busy and unable to respond to storage commands, resulting in command timeouts.
  • Consequently, guest operating systems issue SCSI resets from the guest layer to clear pending reservations and I/O in an attempt to restore the device to an initial power-on state.
  • This behavior stems from a third-party issue.

Resolution

Contact the storage hardware vendor to analyze controller logs and diagnose the underlying cause of the persistent disk busyness.

Additional Information

For detailed definitions of common NVMe opcodes and status codes, refer KB: NVMe OpCodes and Status Definitions

Powered by Wolken Software