NVMe SMART Errors Failing to Trigger vSAN Disk Isolation (vSAN OSA/ESA)

Products

• VMware vSAN

Issue/Introduction

NVMe storage devices may report critical S.M.A.R.T. health warnings (specifically regarding power-loss protection) without triggering automatic disk evacuation or isolation by the vSAN health management service. This document explains the behavior and appropriate manual remediation steps.

Symptoms

• Hardware management consoles (e.g., iDRAC, iLO, XCC) report hardware drive failures.

• vSAN Health Service may report the disk as healthy despite hardware-level alerts.

• You may experience a spike in write latency in the environment (<=50 ms)

• You may or may not see a reported disk issue in vSAN Health (Physical Disk Health - Operation Health) , however the disk remains mounted as seen in Disk Management.
  

 

• The vobd.log records specific critical health warnings, such as: [vob.vsan.lsom.backupfailednvmediskhealthcriticalwarning] NVMe critical health warning for disk <disk_id> is: The disk's backup device has failed.
• Diagnostic output (e.g., esxcli nvme device log smart get) indicates: Volatile Memory Backup Device Failure: true

[root@ESXi:~] esxcli nvme device log smart get -A vmhba1
SMART And Health Info:
   Available Spare Space Below Threshold: false
   Temperature Warning: false
   NVM Subsystem Reliability Degradation: false
   Read Only Mode: false
   Volatile Memory Backup Device Failure: true
   Composite Temperature: 306 K
   Available Spare: 100 %
   Available Spare Threshold: 10 %
   Percentage Used: 0 %
   Data Units Read: 0x60ea0528
   Data Units Written: 0x2f8fbda9
   Host Read Commands: 0x27f7a927fb
   Host Write Commands: 0x12f8084edf
   Controller Busy Time: 0x13522
   Power Cycles: 0x1a
   Power On Hours: 0x3f91
   Unsafe Shutdowns: 0x9
   Media Errors: 0x0
   Number of Error Info Log Entries: 0x2c
   Warning Composite Temperature Time: 0 Mins
   Critical Composite Temperature Time: 0 Mins
   Temperature Sensor 1: 319 K
   Temperature Sensor 2: 309 K
   Temperature Sensor 3: 0 K
   Temperature Sensor 4: 0 K
   Temperature Sensor 5: 0 K
   Temperature Sensor 6: 0 K
   Temperature Sensor 7: 0 K
   Temperature Sensor 8: 0 K

• In the vsandevicemonitord.log:

2026-02-26T20:00:06Z In(14) vsandevicemonitord[2100825]: [70509974144]: WARNING - NVMe critical health warning for disk t10.NVMe____Dell_NVMe_ISE_PS1030_MU_U.2_6.4TB_______############# is: 'The disk's backup device has failed'.
2026-02-26T20:10:07Z In(14) vsandevicemonitord[2100825]: [70509974144]: WARNING - NVMe critical health warning for disk t10.NVMe____Dell_NVMe_ISE_PS1030_MU_U.2_6.4TB_______############# is: 'The disk's backup device has failed'.

• Output of the CLI : esxcli nvme device log smart get -A vmhba1
[root@ESXi:~] esxcli storage core device smart get -d t10.NVMe____Dell_NVMe_ISE_PS1030_MU_U.2_6.4TB_______#####################
Parameter                 Value    Threshold  Worst  Raw
------------------------  -------  ---------  -----  ---
Health Status             WARNING  N/A        N/A    N/A
Power-on Hours            16273    N/A        N/A    N/A
Power Cycle Count         26       N/A        N/A    N/A
Reallocated Sector Count  0        90         N/A    N/A
Drive Temperature         33       75         N/A    N/A

Environment

• VMware vSAN (OSA & ESA).

Cause

vSAN does not automatically isolate NVMe drives based solely on vendor-specific S.M.A.R.T. metrics (such as a volatile memory backup failure). These metrics currently lack industry-wide standardization, and automated responses risk creating false-positive failures that cause unnecessary cluster-wide data rebuilds and performance degradation. vSAN logic specifically monitors for NVM Subsystem Reliability Degradation flags.

Resolution

1. Validate Hardware Health: Run the following command on the affected host to confirm the specific SMART attribute: esxcli nvme device log smart get -A <vmhba_id> Verify if Volatile Memory Backup Device Failure is set to true.
2. Engage Hardware Vendor: Contact the hardware vendor to report the physical hardware fault (failed power-loss protection/capacitor). Request a formal diagnostic and disk replacement if confirmed.
3. Proactive Monitoring: Configure custom vCenter Alarms for the specific NVMe VOB alerts to ensure proactive notification:
   • vob.vsan.lsom.backupfailednvmediskhealthcriticalwarning
   • Refer to Broadcom KB Enabling vSAN alerts for NVMe SMART data in vCenter for step-by-step instructions on setting these alarms.
4. Perform Disk Replacement:
   • Place the host in Maintenance Mode using "Ensure Accessibility."
   • Remove the failed disk from the vSAN disk group/storage pool.
   • Replace the physical hardware.
   • Claim the new device into the vSAN storage pool.

Additional Information

Enhanced Intelligence in vSAN 8 U3 and VMware Cloud Foundation 5.2

vSAN NVMe disk report read only critical warning

Determination of physical Disk Location on Host

Commands to Display S.M.A.R.T. Data for NVMe Devices in ESXi

ESXi S.M.A.R.T. health monitoring for hard drives
 

NVMe Specification - 2.0-2024.08.05 - Page 108:

Critical Warning (CWARN): This field indicates critical warnings for the Controller.
The value of this field shall indicate the value of the Critical Warning field in the Controller’s SMART / Health Information log page. 
Volatile Memory Backup Failed (VMBF): This bit shall indicate the same value as the Volatile Memory Backup Failed (VMBF) bit (i.e., bit 4) in the Critical Warning field in the Controller’s SMART / Health Information log page.