vSAN capacity disk reports SMART impending failure and permanent device error

Products

• VMware vSAN

Issue/Introduction

Symptoms

• A capacity disk in a vSAN OSA (Hybrid) disk group is reporting an impending failure and was marked first offline, then as a Permanent Error (PERM).
• Disk mapping output (vdq -iH):

   DiskMapping[0]:
           SSD:  naa.50000######b800
            MD:  naa.50000######aba5
            MD:  naa.50000######ac15
            MD:  naa.50000######abc1
            MD:  naa.50000######ab51
            MD:  naa.50000######ac35

• All the disks in disk groups may show below state and reason - 
• • "State"    : "Ineligible for use by VSAN",
  • "Reason"   : "Not mounted on this host",

         Evidence 

[root@ESXi:~] vdq -q
[
  {
      DiskResult[7]:
            Name:  naa.50025######847
        VSANUUID:  52####bb-####-####-####-48########53
           State:  Ineligible for use by VSAN
          Reason:  Not mounted on this host
 StoragePoolState: Ineligible for use by Storage Pool
 StoragePoolReason:Disk in use by disk group
          IsSSD?:  1
IsCapacityFlash?:  1
          IsPDL?:  0
        Size(MB):  3662830
      FormatType:  512e
   IsVsanDirectDisk?:  0

  },

• Disk Overall State is as below

UUID: 52####bb-####-####-####-48########53:
   Name: naa.50025######847
   Owner: N/A
   Version: -1
   Disk Group: N/A
   Disk Tier: Unknown
   SSD: true
   In Cmmds: false
   In Vsi: false
   Fault Domain: N/A
   Model: SAMSUNG #####
   Encryption: false
   Compression: false
   Deduplication: false
   Dedup Ratio: N/A
   Overall Health: red(EVACUATED,SMART_DISK_FAILURES)
   Metadata Health: green
   Operational Health: yellow
   Congestion Health:
      CongestionHealth:
         State: green
         Congestion Value: 0
         Congestion Area: None
         All Congestion Fields:
   Space Health:

• Log entries show the device transitioned to PERM error in/var/log/vmkernel.log: 

2025-10-01T16:46:19.614Z In(182) vmkernel: cpu56:2100228)LSOM: LSOMLogDiskEvent:8418: Disk Event permanent error for MD 52###9fd-####-####-####-d7e####2fcf3 (naa.50000######abc5:2)
2025-10-01T16:46:19.614Z Wa(180) vmkwarning: cpu56:2100228)WARNING: LSOM: LSOMEventNotify:8891: vSAN device 52###9fd-####-####-####-d7e####2fcf3  is under permanent error.

• SMART data confirms the disk is in impending failure state:

esxcli storage core device smart get -d naa.50000######abc5

SMART Data for Disk : naa.50000######abc5
Parameter                       Value  Threshold Worst  Raw
-----------------------------------------------------------
Health Status                    IMPENDING FAILURE       N/A     N/A     N/A
Write Error Count                0       N/A     N/A     N/A
Read Error Count                 504     N/A     N/A     N/A
Power Cycle Count                50      N/A     N/A     N/A
Drive Temperature                19      N/A     N/A     N/A
------------------------------------------------------------

Environment

• VMware vSAN 8.x

• VMware vSAN OSA (Hybrid)

Cause

The capacity disk naa.50000######abc5 has suffered a physical hardware failure, evidenced by SMART impending failure alerts and repeated unrecoverable read errors (bad sectors). After exhausting all retry thresholds, vSAN marked the device as permanently failed (PERM) and automatically initiated data evacuation to healthy drives to prevent data loss.

• The disk is experiencing read failures due to physical bad blocks (Sense Key [0x3] MEDIUM ERROR). Consequently, read commands (Cmd 0x28) are exhausting their retries, timing out, and causing the host to abort the operations (Host Status [0x5] ABORT).

2025-09-30T16:26:12.316Z Wa(180) vmkwarning: cpu1:2098534)WARNING: HPP: HppScsiThrottleLogForDevice:585: Cmd 0x28 (0x45b######b80, 0) to dev "naa.50000######abc5" on path "vmhba0:C0:T5:L0" Failed:
2025-09-30T16:26:12.316Z Wa(180) vmkwarning: cpu1:2098534)WARNING: HPP: HppScsiThrottleLogForDevice:593: Error status H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x1. hppAction = 1
2025-09-30T16:26:12.316Z In(182) vmkernel: cpu1:2098534)ScsiDeviceIO: 4686: Cmd(0x45b######b80) 0x28, CmdSN 0xb860211 from world 0 to dev "naa.50000######abc5" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x1 Medium Error, LBA: 102####576

• I/O errors on device naa.50000######abc5 are causing read failures across the disk, rendering even the MBR/GPT partition tables unreadable.

2025-09-30T16:26:49.147Z In(182) vmkernel: cpu2:2116277 opID=da3a69f2)Partition: 477: Failed read for "naa.50000######abc5": I/O error
2025-09-30T16:26:49.147Z In(182) vmkernel: cpu2:2116277 opID=da3a69f2)Partition: 1205: Failed to read protective mbr on "naa.50000######abc5" : I/O error
2025-09-30T16:26:49.147Z Wa(180) vmkwarning: cpu2:2116277 opID=da3a69f2)WARNING: Partition: 1387: Partition table read from device naa.50000######abc5 failed: I/O error
2025-09-30T16:26:49.147Z In(182) vmkernel: cpu2:2116277 opID=da3a69f2)ScsiDeviceIO: 6478: Command 0x1a (CmdSN 0x36###49, World 0) to device naa.50000######abc5 timed out: expiry time occurs 3ms in the past
2025-09-30T16:26:49.147Z Wa(180) vmkwarning: cpu2:2116277 opID=da3a69f2)WARNING: ScsiDeviceIO: 6723: Failed to issue command (0x1a) on device naa.50000######abc5: Timeout

• vSAN detected I/O errors and marked the device offline to resync its data to healthy drives; it will remain out of service until an unmount-mount operation is complete

2025-09-30T16:27:10.948Z In(182) vmkernel: cpu43:16728488)PLOG: PLOGHandleTransientErrorInt:5530: Throttled: Device: 52###9fd-####-####-####-d7e######cf3 will be out of service until unmount-mount operation is complete.
2025-09-30T16:27:10.948Z Wa(180) vmkwarning: cpu43:16728488)WARNING: PLOG: PLOGHandleTransientErrorInt:5612: vSAN device 52###9fd-####-####-####-d7e######cf3 is being repaired due to I/O failures, and will be out of service until the repair is complete. If the devi$
2025-09-30T16:27:10.948Z In(182) vmkernel: cpu43:16728488)LSOMCommon: IORETRYCompleteIO:469: Throttled:  0x45e#####7900 IO type 16648 (READ) isOrdered:NO isSplit:YES isEncr:YES since 60001 msec status Maximum kernel-level retries exceeded

• The disk is exhibiting intermittent responsiveness that is highly typical of a failing drive. This is evidenced by severe latency spikes, where response times suddenly jump from roughly 5 ms (4994 µs) to nearly 1 second (956638 µs) before dropping back down.

2025-09-30T16:41:43.666Z In(182) vmkernel: cpu55:2100228)LSOM: LSOMNamespaceCheckLatency:398: Throttled: Latency 523bd9fd-####-####-####-d7e######cf3 1 18:26:##:##:#:#:0:0:1
2025-09-30T16:41:43.666Z In(182) vmkernel: cpu55:2100228)LSOM: LSOMNamespaceCheckLatency:428: Throttled: LatencyCum 523bd9fd-####-####-####-d7e######fcf3 1 31###81:242##93:22##989:491##96:32##23:14##6:28:1:3

2025-09-30T16:41:53.265Z Wa(180) vmkwarning: cpu35:2098536)WARNING: ScsiDeviceIO: 1780: Device naa.50000######abc5 performance has deteriorated. I/O latency increased from average value of 4994 microseconds to 956638 microseconds.
2025-09-30T16:41:53.271Z In(182) vmkernel: cpu5:2098530)ScsiDeviceIO: 1780: Device naa.50000######abc5 performance has improved. I/O latency reduced from 956638 microseconds to 13441 microseconds.

• The device has exceeded its internal retry limits by reaching the maximum repair threshold of 3. As a result, it has been marked with a permanent error, and vSAN now considers the disk to be permanently failed.

2025-10-01T16:46:19.614Z In(182) vmkernel: cpu29:17011586)PLOG: PLOGHandleTransientErrorInt:5549: Repair threshold (3) for device: 52####fd-####-####-####-d7e######cf3 has been reached and will be marked as PERM error
2025-10-01T16:46:19.614Z Wa(180) vmkwarning: cpu1:2099350)WARNING: PLOG: PLOGPropagateErrorInt:4915: vSAN device 523bd9fd-####-####-####-d7e######cf3 is under permanent error.
2025-10-01T16:46:19.614Z In(182) vmkernel: cpu56:2100228)LSOM: LSOMLogDiskEvent:8418: Disk Event permanent error for MD 52####fd-####-####-####-d7e######cf3 (naa.50000######abc5:2)
2025-10-01T16:46:19.614Z Wa(180) vmkwarning: cpu56:2100228)WARNING: LSOM: LSOMEventNotify:8891: vSAN device 523bd9fd-####-####-####-d7e######cf3 is under permanent error.

• The vsandevicemonitord.log confirms an impending SMART failure on the disk.

2026-05-26T17:27:59Z In(14) vsandevicemonitord[2099813]: [281624289920]: WARNING - SMART health status for disk naa.50025######847 is 2.

2026-05-26T17:27:59Z In(14) vsandevicemonitord[2099813]: [281712289536]: Unmounting the disk/dg: naa.50025######835 with evacReason: SmartDiskFailures

2026-05-26T17:37:59Z In(14) vsandevicemonitord[2099813]: [281624289920]: Device naa.50025######847 state is DISK_UNHEALTHY_LATENCY_SMART

2026-05-26T17:38:00Z In(14) vsandevicemonitord[2099813]: [281624289920]: Cannot auto remediate disk naa.50025######847 for reason SmartDiskFailures, a remediation is already in progress on this host.
2026-05-26T17:38:21Z In(14) vsandevicemonitord[2099813]: [281712289536]: Diskgroup/disk naa.50025######847 unmounted successfully for evacReason: SmartDiskFailures

• /var/run/log/vobd.log - vSAN generated alerts indicating - Impending SMART failure, Device evacuation and unmount and Propagated disk group errors.

2025-10-02T05:08:54.611Z In(14) vobd[2097763]:  [vSANCorrelator] 4657######206us: [esx.problem.vob.vsan.lsom.devicewithsmartfailure] vSAN device naa.50000######abc5 smart health status is impending failure. It will be evacuated and unmounted, consider replacing it.

• /var/run/log/vmkernel.log - Kernel logs confirm - Propagation of unhealthy state to all member disks and Disk group unmount operations completed successfully

2026-05-26T17:27:59.483Z Wa(180) vmkwarning: cpu25:2098607)WARNING: PLOG: PLOGPropagateError:5012: DDP: Propagating unhealthy(noevac) state from original device 52####bb-####-####-####-486######153

2026-05-26T17:27:59.483Z Wa(180) vmkwarning: cpu25:2098607)WARNING: PLOG: PLOGPropagateErrorInt:4920: vSAN device 52####bb-####-####-####-486######153 is unhealthy.

2026-05-26T17:27:59.483Z In(182) vmkernel: cpu86:2099085)LSOM: LSOMLogDiskEvent:8418: Disk Event unhealthyNoEvac for MD 52####bb-####-####-####-486######153 (naa.50025######847:2)

• /var/run/log/vsanmgmt.log - vSAN management logs confirm - Disk no longer present in PLOG/CMMDS and Device state recorded as DISK_UNHEALTHY_LATENCY_SMART

2026-05-27T09:38:55.825Z In(14) vsand[2100008]: [opID=2100008 VsanLsomHealth::checkDiskState] Got devResState from devsTelemetry for 52####bb-####-####-####-486######153: DISK_UNHEALTHY_LATENCY_SMART

2026-05-27T09:38:55.826Z Wa(12) vsand[2100008]: [opID=2100008 VsanHealthSystemImpl::_QueryPhysicalDiskHealthSummary] Failed to get disk 52####bb-####-####-####-486######153 rebalance result : (vim.fault.VsanFault) {
2026-05-27T09:38:55.826Z Wa(12)[+] vsand[2100008]:   faultMessage = (vmodl.LocalizableMessage) [
2026-05-27T09:38:55.826Z Wa(12)[+] vsand[2100008]:     (vmodl.LocalizableMessage) {
2026-05-27T09:38:55.826Z Wa(12)[+] vsand[2100008]:       key = 'com.vmware.vsan.health.msg.invaliddiskuuid.error',
2026-05-27T09:38:55.826Z Wa(12)[+] vsand[2100008]:       message = 'Got invalid disk uuid for vSAN rebalance status'
2026-05-27T09:38:55.826Z Wa(12)[+] vsand[2100008]:     }
2026-05-27T09:38:55.826Z Wa(12)[+] vsand[2100008]:   ]
2026-05-27T09:38:55.826Z Wa(12)[+] vsand[2100008]: }

Resolution

Important: Ensure all data evacuation tasks are 100% complete before physically removing any drives.

Scenario 1: Standard vSAN Cluster (No Deduplication & Compression)

1. Place the affected ESXi host into Maintenance Mode (Select "Ensure Accessibility" or "Full Data Migration").
2. Monitor vCenter tasks and wait for data evacuation to finish.
3. Physically remove the failing disk from the host.
4. Insert the replacement disk into the server.
5. Add the new disk to the existing vSAN disk group in vSphere Client.
6. Exit the host from Maintenance Mode.
7. Monitor vSAN resync to ensure cluster health returns to "Healthy."

Scenario 2: vSAN Cluster with Deduplication & Compression Enabled

1. Place the affected ESXi host into Maintenance Mode (Select "Ensure Accessibility" or "Full Data Migration").
2. Monitor vCenter tasks and wait for data evacuation to finish.
3. In vSphere Client, remove the entire disk group containing the failing disk.
4. Physically remove the failing disk from the host.
5. Insert the replacement disk into the server.
6. Recreate the disk group using the cache drive and all associated capacity drives (including the new replacement).
7. Exit the host from Maintenance Mode.
8. Monitor vSAN resync to ensure cluster health returns to "Healthy."