vSAN ESA Cluster High Latency and VM Unresponsiveness due to NVMe Hardware Failure

Products

• VMware vSAN

Issue/Introduction

Symptoms

• Virtual Machine(s)  becomes unresponsive or inaccessible on the vCenter Server.
• VM guest logs (vmx) show heartbeat timeouts and SCSI command aborts:

PVSCSI: scsi3:0: aborting cmd 0x388
Tools: Tools heartbeat timeout

• VMkernel logs report severe storage performance deterioration:

WARNING: StorageDeviceIO: Device performance has deteriorated. I/O latency increased from average value of 66 microseconds to 36075 microseconds.

• vsish stats report latency buckets exceeding 10 seconds.

Histogram Bucket Array: {
26797009957        (<=                 10)
0                  (<=           10000000)
10714              (>            10000000) > 10 seconds.

• vSAN Device Monitor (vsandevicemonitord) logs a critical health warning:

WARNING - NVMe critical health warning for disk [DeviceID] is: 'The disk's backup device has failed'.

Environment

• VMware vSAN 8.x (ESA)
• Dell Hardware

Cause

• The issue is caused by a hardware-level "soft failure" on a physical NVMe drive
• In these instances, the drive reports a SMART error ("backup device failure") and high latency but remains partially functional.
• Because the failure is not detected by vSAN in this version for vSAN Dying Disk Handling (DDH) to automatically mark it OFFLINE, the extreme latency propagates across the vSAN cluster, impacting overall performance and VM stability.

Resolution

1. Coordinate Hardware Replacement: Contact the hardware vendor (e.g., Dell) to replace the NVMe drive reporting the physical failure.
2. Manual Intervention: If DDH has not automatically isolated the drive, manually evacuate the affected ESXi host using Full Data Migration to prevent further latency propagation.
3. Proactive Mitigation (Upgrade): Upgrade to VMware Cloud Foundation (VCF) 9.0 or higher. VCF 9.0 introduces enhanced latency handling for vSAN ESA, engineered to detect and isolate these types of "soft failures"  reported by SMART more effectively without cluster-wide impact. - VCF 9.0 - What is New