ESXi Host Becomes Unresponsive During Maintenance Mode Due to SEsparse Deadlock caused by Storage Latency

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms: 

The primary product impacted is VMware vSphere ESXi (specifically version 8.0 Update 3) within the Storage component.

Users may observe that an ESXi host becomes completely unresponsive or enters a "Not Responding" state in vCenter Server after attempting to enter Maintenance Mode. Migration tasks (vMotion) may fail or hang indefinitely.

Exact Error Messages:

• ALERT: hostd performance has degraded due to high system latency
  
  
• reason = "Too many outstanding operations"
  
  
• cpu130:2262810)PVSCSI: 2769: scsi0:0: SCSI ABORT ctx=0x37a
cpu130:2262810)nfnic: <1>: INFO: fnic_taskMgmt: 2129: TaskMgmt abort sc->cdb: 0x2a
cpu130:2262810)nfnic: <1>: INFO: fnic_abort_cmd: 3803: Abort cmd called for Tag: 0x1f0  issued time: 180040 ms CMD_STATE: FNIC_IOREQ_CMD_PENDING CDB Opcode: 0x2a  sc:0x45de9900d880 flags: 0x3 lun: 140 target: 0x2f0061
cpu130:2262810)WARNING: nfnic: <1>: fnic_abort_cmd: 3818: Abort for cmd tag: 0x1f0 in pending state
cpu130:2262810)nfnic: <1>: INFO: fnic_fcpio_icmnd_cmpl_handler: 1803: io_req: 0x45de8cc03680 sc: 0x45de9900d880 tag: 0x1f0 CMD_FLAGS: 0x53 CMD_STATE: FNIC_IOREQ_ABTS_PENDING ABTS pending hdr status: FCPIO_ABORTED scsi_status: 0x$
cpu130:2262810)nfnic: <1>: INFO: fnic_fcpio_itmf_cmpl_handler: 2328: fcpio hdr status: FCPIO_SUCCESS
cpu130:2262810)nfnic: <1>: INFO: fnic_fcpio_itmf_cmpl_handler: 2399: io_req: 0x45de8cc03680 sc: 0x45de9900d880 id: 0x1f0 CMD_FLAGS: 0x73 CMD_STATE: FNIC_IOREQ_ABTS_PENDINGhdr status: FCPIO_SUCCESS ABTS cmpl received
  
  
• cpu32:2099201)ScsiDeviceIO: 4644: Cmd(0x45bb52bc4480) 0x2a, CmdSN 0x3ac from world 2806123 to dev "naa.##############################" failed H:0x5 D:0x0 P:0x0

Environment

• Product: VMware ESXi 8.0 Update 3 (Build 25067014)
• Hardware: Cisco Systems Inc UCSX-215C-M8 (Cisco UCS VIC Fnic Controller)
• Storage: Fibre Channel (FC) Storage
• Driver: nfnic version 5.0.0.42-1vmw.803.0.0.24022510
• Disk Type: SEsparse virtual disks

Cause

The issue is caused by a filesystem deadlock involving an RWMutex on an SEsparse virtual disk during a background extension operation.

Technical analysis of core dumps reveals that an internal asynchronous I/O helper thread attempts to extend the physical allocation of an SEsparse disk. This thread requires an exclusive write lock but is blocked because other worker threads hold shared read slices while performing file attribute inquiries. Due to underlying storage latency and SCSI aborts (H:0x5, H:0x8) reported by the nfnic driver, these threads cannot finalize their context, resulting in a permanent deadlock that causes hostd to become sluggish and eventually hang.

Resolution

1. Update Drivers and Firmware:
   

• Verify the current nfnic driver and firmware levels against the VMware Compatibility Guide (HCL).
• Update the Cisco UCS VIC nfnic driver and corresponding hardware firmware to the recommended versions to mitigate SCSI command timeouts and aborts.

1. Storage Subsystem Audit:
   
   

   • Engage CISCO UCS support to investigate potential frame drops or high latency on the Fibre Channel fabric to specific host/blade.
   • Also engage Storage team to validate the storage from Latency perspective