Disk group latency in vSAN cluster due to NVMe stuck I/O
search cancel

Disk group latency in vSAN cluster due to NVMe stuck I/O

book

Article ID: 413903

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

Symptoms:

  • Disk group latency observed on a host in a vSAN cluster.
  • Latency spike is observed on a host/cluster level for a moment and back to normal.
  • I/O timeout event observed for a NVMe disk with stuck I/O reported on a hosts in the vmkwarning.log file:

2025-09-08T11:37:15.079Z Wa(180) vmkwarning: cpu6:2098027)WARNING: PLOG: PLOG_DeviceHandleIOTimeOut:8792: vSAN device ########-####-####-####-############ detected I/O timeout error. This may lead to stuck I/O.

  • Heartbeat timeout for the VM Namespace reported:

2025-09-08T11:37:15.514Z In(14) vobd[2098145]:  [vmfsCorrelator] 1968738685513us: [esx.problem.vmfs.heartbeat.timedout] ########-########-####-############ ########-####-####-####-############

  • Heartbeat recovery events for the VM Namespaces reported.This is where the driver would have got a response from the device post resets that the driver would have performed.The recovery of the vSAN objects and I/O redirection occurred to minimize effect of these issues on the VMs and their services:
2025-09-08T11:37:37.098Z In(14) vobd[2098146]:  [vmfsCorrelator] 1149814230097us: [esx.problem.vmfs.heartbeat.recovered] Reclaimed heartbeat for volume ########-########-####-############ (########-####-####-####-############): [Timeout] [HB state abcdef02 offset 3702784 gen 81 stampUS 1149814222874 uuid ##########-########-####-############ jrnl <FB 7> drv 24.82]

 

Environment

VMware vSAN 8.x

VMware vSAN 9.x

Cause

The stuck I/O events on the NVMe  is caused due to the underlying device not responding to I/Os or aborts.This led to the driver issuing reset, which failed, consequently leading to the device being marked offline by the driver.Telemetry data is typically unavailable when a disk is in offline state.

  • PSA issued aborts to the device:
025-09-08T11:37:16.423Z Wa(180) vmkwarning: cpu2:2098523)WARNING: NVMeUtil: 151: Error on Cmd(0x45c37f30c840) 0x2, CmdSN 0x1b3c6a89 from world 0 to component "t10.NVMe____Dell_Ent_NVMe_CM6_RI_3.84TB_____________###############"  H:0x6 D:0x0 P:0x0

  • Controller reset:
025-09-08T11:37:33.082Z Wa(180) vmkwarning: cpu24:10045396)WARNING: NVMEIO:4011 Controller 256 in state 8 or in recovery mode, bail out.

  • Device repair event:
2025-09-08T11:37:36.414Z Wa(180) vmkwarning: cpu0:2099871)WARNING: PLOG: PLOGHandleTransientErrorInt:5612: vSAN device ########-####-####-####-############ is being repaired due to I/O failures, and will be out of service until the repair is complete. If the device$

2025-09-08T11:37:36.414Z In(14) vobd[2098145]:  [vSANCorrelator] 1968806263750us: [esx.problem.vob.vsan.lsom.devicerepair] Device ########-####-####-####-############ is in offline state and is getting repaired.

  • LSOM event showing the disk has gone offline :
2025-09-08T11:37:36.424Z Wa(180) vmkwarning: cpu49:2099615)WARNING: LSOM: LSOMEventNotify:9026: vSAN device ########-####-####-####-############ has gone offline.
  • Task mgmt aborts also stuck:
2025-09-08T11:37:45.082Z In(182) vmkernel: cpu50:10045441)StorageDeviceIO: 5608: Task mgmt request issued to device t10.NVMe____Dell_Ent_NVMe_CM6_RI_3.84TB_____________############### is stuck (WorldID 0, CmdSN 1b3c69f0). Issuing yellow notification to the application

  • Device under transient error processing:
2025-09-08T11:37:45.082Z Wa(180) vmkwarning: cpu4:2098027)WARNING: PLOG: PLOG_DeviceHandleIOTimeOut:8783: Device ########-####-####-####-############ is under transient error processing

  • APD event:
2025-09-08T11:37:55.081Z In(14) vmkernel: cpu42:2098028)StorageApdHandlerEv: 106: Device or filesystem with identifier  [t10.NVMe____Dell_Ent_NVMe_CM6_RI_3.84TB_____________###############] has entered the All Paths Down state.

  • Device in PDL state:
2025-09-08T11:37:55.082Z In(14) vmkwarning: cpu10:2098039)WARNING: StorageDevice: 5587:  Device :t10.NVMe____Dell_Ent_NVMe_CM6_RI_3.84TB_____________############### has been removed or is permanently inaccessible.
2025-09-08T11:37:57.091Z In(14) vobd[2098145]:  [vSANCorrelator] 1968780261049us: [vob.vsan.pdl.offline] vSAN device ########-####-####-####-############ has gone offline.

  • LSOM event indicating disk not found:
2025-09-08T11:37:57.091Z Wa(180) vmkwarning: cpu20:2099615)WARNING: LSOM: LSOMPostDiskEvent:3788: Throttled: Unable to post disk event for ########-####-####-####-############: Not found

  • Failed to read the controller :
2025-09-08T11:38:19.975Z Wa(180) vmkwarning: cpu50:2098524)WARNING: NVMEDEV:8343 Failed to enable controller 256, status: Device is permanently unavailable

 
 

Resolution

Follow these steps to recover the host and investigate the hardware failure:

  1. Migrate virtual machine workloads to other hosts in the cluster to ensure service continuity.
  2. Place the affected host into Maintenance Mode.
  3. Perform a warm boot (power cycle) of the host via the server management interface (e.g., iDRAC, iLO).
  4. Engage the hardware vendor to investigate potential hardware or firmware-related causes on the identified NVMe device.
  5. Collect vm-support bundles post-reboot for vendor analysis.

For additional assistance, see Contact Broadcom Support.

To receive updates on fix status for similar issues, Subscribe to this Article.

Powered by Wolken Software