VM Hangs and VSCSI Resets Due to Unresponsive NVMe Fibre Channel Storage Target

Products

• VMware vSphere ESXi

Issue/Introduction

Virtual machines experience sudden unresponsive states or hangs accompanied by underlying Virtual SCSI (VSCSI) resets in the VMware vSphere environment.

Log analysis and diagnostic outputs indicate that I/O commands sent to the storage subsystem are permanently blocked below the host bus adapter (HBA) layer, resulting in an accumulation of outstanding commands across multiple paths without any physical link down events or hardware errors registered on the host side:

Refer to outstanding commands in /usr/lib/vmware/vmkmgmt_keyval/vmkmgmt_keyval -a

NVMe Host Device Name: vmhba69
Total number of outstanding commands: 16
NVMe Host Device Name: vmhba67
Total number of outstanding commands: 16
NVMe Host Device Name: vmhba68
Total number of outstanding commands: 17
NVMe Host Device Name: vmhba64
Total number of outstanding commands: 17
NVMe Host Device Name: vmhba65
Total number of outstanding commands: 16

Additionally, the storage subsystem adapter captures high-frequency fabric Registered State Change Notifications (RSCNs) approximately every 30 minutes, causing persistent disruptions to the active I/O pipeline:

2026-05-10 02:53:17.054 [vmhba64] RSCN received for PID <REDACTED_SECRET>
2026-05-10 02:53:17.054 [vmhba64] RSCN received for PID <REDACTED_SECRET>
2026-05-10 02:53:17.095 [vmhba64] RSCN received for PID <REDACTED_SECRET>
2026-05-10 03:22:45.773 [vmhba64] RSCN received for PID <REDACTED_SECRET>

Environment

VMware vSphere ESXi 8.x

Cause

The target storage array fails to return or complete outstanding command frames dispatched by the host bus adapter driver, which is further exacerbated by fabric-wide instability and port transitions triggering excessive Registered State Change Notifications.

Resolution

To resolve this issue, perform the following troubleshooting and configuration checks on the physical storage area network and storage array:

1. Engage the storage hardware vendor to analyze array-side controller and target logs to determine why incoming NVMe-FC command frames are unacknowledged or dropped without status responses.

2. Investigate the physical Fibre Channel switch infrastructure and fabric name servers to isolate the root cause of the high-frequency port transitions causing the 30-minute RSCN intervals.

3. Review and align the SAN fabric architecture with recommended design practices, ensuring correct implementation of single-initiator single-target zoning layouts to minimize baseline RSCN transmission overhead.