vSAN ESA drive automatically unmounts with "StoragePoolDiskType_Unknown" status and "vSAN physical disk alarm Operation" warning

Products

• VMware vSAN

Issue/Introduction

Symptoms:

In a vSAN Express Storage Architecture (ESA) cluster, you observe the following symptoms:

• An alarm is triggered in the vSphere Client indicating 'vSAN physical disk alarm Operation'.
  
  

• Under vSAN Skyline Health, operational health warning is reported as unhealthy for a disk on a specific node
  
  

• Under the Disk Management view, the affected disk shows as unmounted from the storage pool.
  
  

• Running the vdq -iH command on the affected host displays the disk under StoragePoolDiskType_Unknown.
  
  vdq -iH
SingleTierDisks:
 "StoragePoolDiskType_Unknown" : [
 "eui.364350305#######################"
 ]
 "singleTier" : [
 "eui. 364#############################",
 "eui. 364#############################",
 "eui. 364#############################",
 "t10.NVMe______Dell Ent_NVMe_CM6_MU_1.6TB____________________################",
 "eui. 364#############################",
 "eui. 364#############################",
 "eui. 364#############################",
 "eui. 364#############################",
 "t10.NVMe______Dell Ent_NVMe_CM6_MU_1.6TB____________________################",
 "eui. 364#############################"
 "eui. 364#############################",
 ]

• The drive may still report as healthy when inspected directly from the hardware console or basic physical status checks

Environment

VMware vSAN 8.x

Cause

This issue occurs due to an underlying physical hardware degradation or failure on the NVMe drive, characterized by media errors logged in its SMART data.

While the drive might not have reached the total failure threshold required to trigger a hardware console alert, vSAN ESA's strict end-to-end checksum validation detected data inconsistencies (checksum mismatch). To safeguard virtual machine data integrity, vSAN's Dying Disk Handling (DDH) mechanism proactively isolated and unmounted the drive from the storage pool.

Cause Validation

The /var/run/log/vmkernel.log reveal explicit checksum mismatches on the primary and secondary mirrors, leading to the device being forced offline:

2026-06-07T16:28:05.9682 Wa (180) vmkwarning: cpu0: 2098761) WARNING: MDR: MDRIOSubCompletionPrimary: 3116: eui.364350305#######################:2:06-splinter-data: Got error on primary mdrCtx 0x45ba28fd6100 flag=0x8 state=0x1 prevState=0x0
2026-06-07T16:28:05.968Z Wa (180) vmkwarning: cpu0:2098761) WARNING: MDR: MDRIOSubCompletionPrimary: 3122: Primary Status=Not initialized Secondary Status=Not initialized
2026-06-07T16:28:05.968Z Wa (180) vmkwarning: cpu0:2098761) WARNING: MDR: MDRIOSubCompletionPrimary: 3140: Retrying read from secondary mirror
2026-06-07T16:28:05.9682 Wa (180) vmkwarning: cpu0: 2098761) WARNING: MDR: MDRIOHeaderVerifyInternal: 1944: [0] Got mismatched cksum ab100000 c42dd7bb len 24080
2026-06-07T16:28:05.9682 Wa (180) vmkwarning: cpu0:2098761) WARNING: MDR: MDRIOHeaderPrint: 1865: [0] : got hdr=0x43466d2ac000 magic=0x14 hash=0 len=24080 Offset=4398046576640 rel=359915520 disk=30505172992 chksum=0xab100000 bmap = 1
2026-06-07T16:28:05.9682 Wa (180) vmkwarning: cpu0:2098761) WARNING: MDR: MDRIOSubCompletionSecondary: 3242: eui.364350305#######################:2:06-splinter-data: Got error on SEC mdrCtx 0x45ba28fd6100 flag=0x8
2026-06-07T16:28:05.968Z Wa (180) vmkwarning: cpu0:2098761) WARNING: MDR: MDRIOSubCompletionSecondary: 3247: state=0x2 prevState=0xl Primary Status=Transient file system condition, suggest retry fecondary Status=Checksum mismatch
2026-06-07T16:28:05.968Z Wa (180) vmkwarning: cpu30:2101524) WARNING: MDR: MDRIOHOfflineDevice:4669: Offlining with IOLayer error 0xbad003a on eui.364350305#######################:2:06-splinter-data handle
2026-06-07T16:28:05.9682 In(182) vmkernel: cpu30:2101524) WOBTREE: IOLayerOfflineDevice: 6570: IOLayerOfflineDevice eui.364350305#######################:2 region 12-splinter-data-mdr-1 status=Checksum mismatch
2026-06-07T16:28:05.9682 In(182) vmkernel: cpu30:2101524) WOBTREE: IOLayer_SetDeviceOffline:8596: OfflineDevice eui.364350305#######################:2 status=Checksum mismatch errType TRANSIENT
2026-06-07T16:28:05.968Z In (182) vmkernel: cpu30:2101524) WOBTREE: IOLayerOfflineDevice: 6570: IOLayerOfflineDevice eui.364350305#######################:2 region 13-splinter-data-mdr-2 status=Checksum mismatch
2026-06-07T16:28:05.9682 In(182) vmkernel: cpu30:2101524) WOBTREE: IOLayer_SetDeviceOffline: 8596: OfflineDevice eui.364350305#######################:2 status=Checksum mismatch errType TRANSIENT
2026-06-07T16:28:05.968Z Wa (180) vmkwarning: cpu30:2101524) WARNING: WOBTREE: clockcache_get_internal: 4400: RETURN_ON_ERROR [195887162/0xbad003a/Checksum mismatch]

The /var/run/log/vobd.log show that vSAN recognized the error and automatically took the device offline for storage pool repair

2026-06-07T16:28:04.9322 In(14) vobd[2098051] : [vSANCorrelator] 16515381351305us: [vob. vsan.dom.singlediskerrorfixed] vSAN detected and fixed a medium or checksum error for component bd8f0869-####-####-####-############ on disk 52b5da8b-####-####-####-############
2026-06-07T16:28:05.968Z In(14) vobd[2098051]: [vSANCorrelator] 16515382387063us: [vob.vsan. 1som. storagepoolrepair] vSAN device 52b5da8b-####-####-####-############ is being repaired due to I/0 failures and will be out of service until the repair is complete.
2026-06-07T16:28:05.968Z In(14) vobd[2098051]: [vSANCorrelator] 16515530564368us: [esx.problem. vob. vsan. lsom. storagepoolrepair] Device 52b5da8b-####-####-####-############ is currently offline and is being repaired.
2026-06-07T16:28:05.9682 In(14) vobd[2098051]: The event ( [esx. problem. vob. vsan. lsom. storagepoolrepair] Device 52b5da8b-####-####-####-############ is currently offline and is being repaired.) was sent immediately to hostd;

Checking the SMART options of the physical drive reveals a non-zero count for Media Errors, confirming hardware degradation:

esxcli nvme device log smart get -A vmhba#
SMART And Health Info:
Available Spare Space Below Threshold: false
Temperature Warning: false
NVM Subsystem Reliability Degradation: false
Read Only Mode: false
Volatile Memory Backup Device Failure: false
Composite Temperature: 300 K
Available Spare: 100 %
Available Spare Threshold: 10 %
Percentage Used: 0 %
Data Units Read: 0x43e8504
Data Units Written: 0x6134338
Host Read Commands: 0x3a81010d
Host Write Commands: 0x13cd8aa5a
Controller Busy Time: 0x223
Power Cycles: 0x30
Power On Hours: 0x4836
Unsafe Shutdowns: 0x23
Media Errors: 0xd
Number of Error Info Log Entries: 0xbcea6a
Warning Composite Temperature Time: 0 Mins
Critical Composite Temperature Time: 0 Mins
Temperature Sensor 1: 300 K
Temperature Sensor 2: 0 K
Temperature Sensor 3: 0 K
Temperature Sensor 4: 0 K
Temperature Sensor 5: 0 K
Temperature Sensor 6: 0 K
Temperature Sensor 7: 0 K
Temperature Sensor 8: 0 K

Resolution

To clear the transient unmounted state and restore cluster health, follow these steps.

Note: Because the drive has a documented history of physical media errors (0xd), this fix may only be temporary. Active monitoring is required.

Step 1: Recover the Current Disk State

1. Log into the vSphere Client.

2. Place the affected ESXi host into Maintenance Mode selecting the Ensure Accessibility data migration option.

3. Navigate to the host Configure tab > vSAN > Disk Management.

4. Locate the unmounted disk from the storage pool, select it, and click Remove to delete it from the storage configuration.

5. Once fully removed, click Add Disks to add the physical disk back into the vSAN ESA storage pool.

6. Verify that the disk status returns to healthy and mounted.

7. Take the ESXi host out of Maintenance Mode.

Step 2: Monitor and Engage Hardware Vendor

1. Monitor the cluster health closely over the next few days.

2. If the drive unmounts again or the 'vSAN physical disk alarm Operation' recurs, it serves as definitive proof of a recurring hardware fault.

3. Contact your hardware vendor immediately to open a support ticket for a physical drive replacement. Provide the captured SMART logs showing the Media Errors count as justification for the replacement.