vSAN cluster performance degradation and storage failures caused by NVMe controller fatal state (State 9)

Products

VMware vSAN

Issue/Introduction

Symptoms:

When this issue occurs, you may experience severe slowness or latency across your vSAN cluster, typically localized to a specific ESXi host. The host may enter a degraded state where disk groups fail to mount or unmount cleanly.

To verify if you are experiencing this specific issue, review the /var/log/vmkernel.log on the affected host. You will observe the following patterns:

1. Repeated I/O read errors and command aborts: The host frantically attempts to re-establish I/O, resulting in millions of repeated error counts and SCSI status codes (H:0x0 D:0x0 P:0x4) that indicate a physical path failure.

2026-07-15T01:44:17.184Z vmkwarning: cpu39:2097278)WARNING: NvmeUtil: 151: Error on Cmd(0x45ddbe449380) 0x2, CmdSN 0x94d93 from world 0 to component "t10.NVMe____Dell_Express_Flash_NVMe_P4800X_375GB_####" H:0x0 D:0x0 P:0x4
2026-07-15T01:46:45.228Z vmkernel: cpu3:2097245)NvmeUtil: 93: Error status on Cmd(0x45b9c2287080) H:0x0 D:0x0 P:0x4 from component t10.NVMe____Dell_Express_Flash_NVMe_P4800X_375GB_#### repeated 13107200 times
2026-07-15T01:46:57.194Z vmkernel: cpu40:2100041 opID=f95c5ace)Partition: 477: Failed read for "t10.NVMe____Dell_Express_Flash_NVMe_P4800X_375GB_####": I/O error

2. Complete unmount failures: Because the drive is physically unresponsive, it returns 0 bytes when the kernel attempts to read it, preventing a clean unmount.

2026-07-14T14:16:24Z : Unable to unmount: Failed to complete read device: /vmfs/devices/disks/t10.NVMe____Dell_Express_Flash_NVMe_P4800X_375GB_#### bytes read: 0 of 512

3. Controllers dropping out of an operational state:

2026-07-14T19:36:49.655Z Wa(180) vmkwarning: cpu36:2100965)WARNING: NVMEMGMT:1073 Controller #### is not in operational state.
2026-07-14T19:38:09.667Z Wa(180) vmkwarning: cpu40:2100965)WARNING: NVMEMGMT:1073 Controller #### is not in operational state.

4. Devices reporting 100% error rates and stuck I/O:

Basic SMART health queries time out, leading to a DISK_UNDER_STUCK_IO state.

2026-07-14T23:39:08.321Z In(182) vmkernel: cpu37:10771308)StorageDeviceIO: 2653: Too many errors observed for device t10.NVMe____Dell_Express_Flash_NVMe_P4500_2.0TB_SFF_#### errPercentage 100
2026-07-06T23:52:58Z In(14) vsandevicemonitord[2099799]: Device eui.#### state is DISK_UNDER_STUCK_IO

Environment

VMware vSAN 8.x
NVMe Storage Controllers

Cause

This issue is caused by a permanent physical hardware failure. The underlying NVMe controller has entered an unrecoverable hardware freeze (reported as State 9), meaning it has become completely unresponsive on the PCIe bus.

/var/run/log/vmkwarning.log :

Wa(180) vmkwarning: cpu58:2097703)WARNING: NVMEIO:4011 Controller 256 in state 9 or in recovery mode, bail out.

This hardware lockup creates a cascading failure in the ESXi software stack:

Command Timeouts: Because the controller is functionally dead, routine administrative commands (such as the Opcode 0x6 Identify command) and read commands (Opcode 0x2) time out and are forcefully aborted by the ESXi kernel.
Unmount Failures: When vSAN attempts to unmount the failing disk group, it must write final metadata and close file descriptors by reading the drive's master block. Because the hardware is frozen, the read operation fails, preventing the kernel from verifying the disk structure.
Infinite Recovery Loop: A state mismatch occurs between the vSAN service and the ESXi storage driver. Logically, vSAN detects the failure and tries to force an unmount. Physically, the low-level PCIe driver has already detached the dead hardware, so the unmount is rejected. vSAN then attempts to re-mount the drive, which fails instantly, triggering the unmount rule again and creating an infinite loop that causes cluster slowness.

Resolution

The underlying physical controller has entered an unrecoverable state, software intervention cannot resolve the physical layer hardware freeze. The faulty hardware must be replaced to restore cluster health.

Perform a cold power cycle (physical power-off/on) of the impacted host. This is required to reset the PCIe bus and the frozen controller hardware.
Engage the hardware vendor to replace the physical NVMe drive or controller, as the issue indicates a permanent hardware failure.
Engage the Broadcom support Contact Broadcom Support