Operational health warning seen on vSAN ESA cluster

Products

VMware vSAN

Issue/Introduction

In a vSAN Express Storage Architecture (ESA) cluster, a physical disk may report as unhealthy or unmounted in the vSphere Client. Attempts to remove the disk from the storage pool via the vCenter UI or command line fail due to metadata corruption, stuck I/O errors, or hardware-level unresponsiveness. This article outlines the procedure to address these persistent removal failures.

Symptoms

vSAN disk displays as "unhealthy" in Disk Management but still mounted.
Skyline Health reports "Physical disk operation health" warnings.
The command vdq -iH shows all the disks as normal and healthy:
However, the disk shows unmounted:

[localhost :~ ] esxcli vsan storagepool list | grep -i cmmds
In CMMDS: true
In CMMDS: true
In CMMDS: true
In CMMDS: false
In CMMDS: true
In CMMDS: true
In CMMDS: true
In CMMDS: true
Log entries in /var/run/log/vsanmgmt.log indicate transient errors while repairing disks:

2026-08-03T07: 27:57.611Z In (14) vsand [2102296] : [opID=2102296 VsanLsomHealth: : checkDiskState] Got devResState from devsTelemetry for disk 5224c6db-####-####-####-##########: DISK REPAIR FAILED
2026-08-03T07:27:57.6112 In (14) vsand [2102296] : [opID=2102296 VsanHealthSystemImpl: : QueryPhysicalDiskHealthSummary] Disk 5224c6db-####-####-####-########## cmmds health status: None , LSOM telemetry status: TRANSIENT_ERROR_REPAIR_FAILED
The SMART statistics for the device shows health as Warning:

esxcli storage core device smart get -d t10.NVMe___Micron_7450_##########___________###############
Parameter Value Threshold Worst Raw
-------------------------------------------------------------------------------------
Health Status WARNING N/A N/A N/A
Power-on Hours 13614 N/A N/A N/A
Power Cycle Count 28 N/A N/A N/A
Reallocated Sector Count 0 90 N/A N/A
Drive Temperature 32 77 N/A N/A

Environment

VMware vSAN 8.x (Express Storage Architecture - ESA)

Cause

The issue typically stems from underlying physical hardware degradation or media errors on the NVMe drive. When the storage controller or NVMe drive cannot complete I/O requests due to media errors (e.g., Unrecoverable Read Errors), vSAN ESA metadata operations fail. This leads to an inability to cleanly remove the device from the storage pool as the vSAN management stack cannot communicate with the disk to update partition or metadata maps.

Resolution

Place Host in Maintenance Mode: Navigate to the vSphere Client, right-click the affected ESXi host, and enter Maintenance Mode selecting the Ensure Accessibility data migration option.
Validate Disk State: Identify the affected device details from the disk management tab on the vCenter.
Attempt Removal: Attempt to remove the disk via the vCenter UI (Disk Management) or esxcli vsan storagepool remove -u <Disk_UUID>.
Hardware Replacement: If the removal command returns metadata errors (e.g., Unable to remove device: Failed to remove disk:<device_name> with error: Failed to update storage pool metadata for disk <device_name> with error: Failed to invalidate disk metadata: <device_name> with exception: Failed to write partition"), the disk is likely experiencing physical media failure. Engage the hardware vendor to physically replace the NVMe drive with a healthy unit.
Automatic Removal: After replacing the hardware, the disk often automatically clears from the storage pool as the faulted physical path is removed.
Recreate Storage Pool: Once the hardware is replaced and the old device is removed from the storage configuration, add the new disk to the storage pool to restore cluster compliance and data redundancy.
Exit Maintenance Mode: Exit Maintenance Mode once the storage pool reports as healthy.

Additional Information

If the issue persists or if errors involve cluster-wide metadata, contact Broadcom Support for further analysis of the storage stack.