vSAN capacity disk reports SMART impending failure and permanent device error
search cancel

vSAN capacity disk reports SMART impending failure and permanent device error

book

Article ID: 412701

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

A capacity disk in a vSAN OSA (Hybrid) disk group is reporting an impending failure and was marked first offline, then as a Permanent Error (PERM).

Disk mapping output (vdq -iH):

   DiskMapping[0]:
           SSD:  naa.50000######b800
            MD:  naa.50000######aba5
            MD:  naa.50000######ac15
            MD:  naa.50000######abc1
            MD:  naa.50000######ab51
            MD:  naa.50000######ac35

All the disks in disk groups may show below state and reason - 

    • "State"    : "Ineligible for use by VSAN",
    • "Reason"   : "Not mounted on this host",

Evidence 

[root@ESXi:~] vdq -q
[
  {
      DiskResult[7]:
            Name:  naa.50025######847
        VSANUUID:  52####bb-####-####-####-48########53
           State:  Ineligible for use by VSAN
          Reason:  Not mounted on this host
 StoragePoolState: Ineligible for use by Storage Pool
 StoragePoolReason:Disk in use by disk group
          IsSSD?:  1
IsCapacityFlash?:  1
          IsPDL?:  0
        Size(MB):  3662830
      FormatType:  512e
   IsVsanDirectDisk?:  0

  },

Disk Overall State is as below

UUID: 52####bb-####-####-####-48########53:
   Name: naa.50025######847
   Owner: N/A
   Version: -1
   Disk Group: N/A
   Disk Tier: Unknown
   SSD: true
   In Cmmds: false
   In Vsi: false
   Fault Domain: N/A
   Model: SAMSUNG #####
   Encryption: false
   Compression: false
   Deduplication: false
   Dedup Ratio: N/A
   Overall Health: red(EVACUATED,SMART_DISK_FAILURES)
   Metadata Health: green
   Operational Health: yellow
   Congestion Health:
      CongestionHealth:
         State: green
         Congestion Value: 0
         Congestion Area: None
         All Congestion Fields:
   Space Health:

Log entries show the device transitioned to PERM error:

/var/log/vmkernel.log 

2025-10-01T16:46:19.614Z In(182) vmkernel: cpu56:2100228)LSOM: LSOMLogDiskEvent:8418: Disk Event permanent error for MD 52###9fd-####-####-####-d7e####2fcf3 (naa.50000######abc5:2)
2025-10-01T16:46:19.614Z Wa(180) vmkwarning: cpu56:2100228)WARNING: LSOM: LSOMEventNotify:8891: vSAN device 52###9fd-####-####-####-d7e####2fcf3  is under permanent error.

SMART data confirms the disk is in impending failure state:

esxcli storage core device smart get -d naa.50000######abc5

SMART Data for Disk : naa.50000######abc5
Parameter                       Value  Threshold Worst  Raw
-----------------------------------------------------------
Health Status                    IMPENDING FAILURE       N/A     N/A     N/A
Write Error Count                0       N/A     N/A     N/A
Read Error Count                 504     N/A     N/A     N/A
Power Cycle Count                50      N/A     N/A     N/A
Drive Temperature                19      N/A     N/A     N/A
------------------------------------------------------------

Environment

  • VMware vSAN 8.x

  • VMware vSAN OSA (Hybrid)

Cause

  • The capacity device naa.50000######abc5 encountered repeated read errors and medium errors (bad sectors).

  • vSAN attempted multiple retries and repair operations, but the retry threshold was exceeded.

  • Device was first marked offline, then escalated to Permanent Error (PERM).

  • SMART monitoring reports the drive in impending failure.

  • vSAN automatically initiated data evacuation to protect against data loss.

Validation

Medium Errors (Read Failures)

  • Sense Key [0x3] MEDIUM ERROR with READ RETRIES EXHAUSTED.
  • This means the drive has physical problems reading certain sectors (bad blocks developing).

Command Failures & Timeouts

  • Cmd 0x28 … Failed: Medium Error and later Host Status [0x5] ABORT.
  • Commands to the device are being aborted due to repeated timeouts.

2025-09-30T16:26:12.316Z Wa(180) vmkwarning: cpu1:2098534)WARNING: HPP: HppScsiThrottleLogForDevice:585: Cmd 0x28 (0x45b######b80, 0) to dev "naa.50000######abc5" on path "vmhba0:C0:T5:L0" Failed:
2025-09-30T16:26:12.316Z Wa(180) vmkwarning: cpu1:2098534)WARNING: HPP: HppScsiThrottleLogForDevice:593: Error status H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x1. hppAction = 1
2025-09-30T16:26:12.316Z In(182) vmkernel: cpu1:2098534)ScsiDeviceIO: 4686: Cmd(0x45b######b80) 0x28, CmdSN 0xb860211 from world 0 to dev "naa.50000######abc5" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x1 Medium Error, LBA: 102####576

I/O Errors on Partition Reads

  • Failed read for naa.50000######abc5: I/O error
  • Even the partition table (protective MBR/GPT) cannot be read properly.

2025-09-30T16:26:49.147Z In(182) vmkernel: cpu2:2116277 opID=da3a69f2)Partition: 477: Failed read for "naa.50000######abc5": I/O error
2025-09-30T16:26:49.147Z In(182) vmkernel: cpu2:2116277 opID=da3a69f2)Partition: 1205: Failed to read protective mbr on "naa.50000######abc5" : I/O error
2025-09-30T16:26:49.147Z Wa(180) vmkwarning: cpu2:2116277 opID=da3a69f2)WARNING: Partition: 1387: Partition table read from device naa.50000######abc5 failed: I/O error
2025-09-30T16:26:49.147Z In(182) vmkernel: cpu2:2116277 opID=da3a69f2)ScsiDeviceIO: 6478: Command 0x1a (CmdSN 0x36###49, World 0) to device naa.50000######abc5 timed out: expiry time occurs 3ms in the past
2025-09-30T16:26:49.147Z Wa(180) vmkwarning: cpu2:2116277 opID=da3a69f2)WARNING: ScsiDeviceIO: 6723: Failed to issue command (0x1a) on device naa.50000######abc5: Timeout

vSAN Disk Repair Process

  • Device will be out of service until unmount-mount operation is complete
  • vSAN device is being repaired due to I/O failures
  • vSAN detects these errors and marks the device offline, trying to repair by resyncing data to other healthy devices.

2025-09-30T16:27:10.948Z In(182) vmkernel: cpu43:16728488)PLOG: PLOGHandleTransientErrorInt:5530: Throttled: Device: 52###9fd-####-####-####-d7e######cf3 will be out of service until unmount-mount operation is complete.
2025-09-30T16:27:10.948Z Wa(180) vmkwarning: cpu43:16728488)WARNING: PLOG: PLOGHandleTransientErrorInt:5612: vSAN device 52###9fd-####-####-####-d7e######cf3 is being repaired due to I/O failures, and will be out of service until the repair is complete. If the devi$
2025-09-30T16:27:10.948Z In(182) vmkernel: cpu43:16728488)LSOMCommon: IORETRYCompleteIO:469: Throttled:  0x45e#####7900 IO type 16648 (READ) isOrdered:NO isSplit:YES isEncr:YES since 60001 msec status Maximum kernel-level retries exceeded

Latency Spikes

  • Latency shot up from ~5 ms (4994 µs) to nearly 1 second (956638 µs).
  • Then reduced again - typical of a failing disk with intermittent responsiveness.

2025-09-30T16:41:43.666Z In(182) vmkernel: cpu55:2100228)LSOM: LSOMNamespaceCheckLatency:398: Throttled: Latency 523bd9fd-####-####-####-d7e######cf3 1 18:26:##:##:#:#:0:0:1
2025-09-30T16:41:43.666Z In(182) vmkernel: cpu55:2100228)LSOM: LSOMNamespaceCheckLatency:428: Throttled: LatencyCum 523bd9fd-####-####-####-d7e######fcf3 1 31###81:242##93:22##989:491##96:32##23:14##6:28:1:3

2025-09-30T16:41:53.265Z Wa(180) vmkwarning: cpu35:2098536)WARNING: ScsiDeviceIO: 1780: Device naa.50000######abc5 performance has deteriorated. I/O latency increased from average value of 4994 microseconds to 956638 microseconds.
2025-09-30T16:41:53.271Z In(182) vmkernel: cpu5:2098530)ScsiDeviceIO: 1780: Device naa.50000######abc5 performance has improved. I/O latency reduced from 956638 microseconds to 13441 microseconds.

Permanent Device Error

  • Repair threshold (3)  reached and will be marked as PERM error
  • Device has exceeded retry limits. vSAN now considers it permanently failed.

2025-10-01T16:46:19.614Z In(182) vmkernel: cpu29:17011586)PLOG: PLOGHandleTransientErrorInt:5549: Repair threshold (3) for device: 52####fd-####-####-####-d7e######cf3 has been reached and will be marked as PERM error
2025-10-01T16:46:19.614Z Wa(180) vmkwarning: cpu1:2099350)WARNING: PLOG: PLOGPropagateErrorInt:4915: vSAN device 523bd9fd-####-####-####-d7e######cf3 is under permanent error.
2025-10-01T16:46:19.614Z In(182) vmkernel: cpu56:2100228)LSOM: LSOMLogDiskEvent:8418: Disk Event permanent error for MD 52####fd-####-####-####-d7e######cf3 (naa.50000######abc5:2)
2025-10-01T16:46:19.614Z Wa(180) vmkwarning: cpu56:2100228)WARNING: LSOM: LSOMEventNotify:8891: vSAN device 523bd9fd-####-####-####-d7e######cf3 is under permanent error.

SMART Impending Failure

  • /var/log/vsandevicemonitord.log - SmartDiskFailures

2026-05-26T17:27:59Z In(14) vsandevicemonitord[2099813]: [281624289920]: WARNING - SMART health status for disk naa.50025######847 is 2.

2026-05-26T17:27:59Z In(14) vsandevicemonitord[2099813]: [281712289536]: Unmounting the disk/dg: naa.50025######835 with evacReason: SmartDiskFailures

2026-05-26T17:37:59Z In(14) vsandevicemonitord[2099813]: [281624289920]: Device naa.50025######847 state is DISK_UNHEALTHY_LATENCY_SMART

2026-05-26T17:38:00Z In(14) vsandevicemonitord[2099813]: [281624289920]: Cannot auto remediate disk naa.50025######847 for reason SmartDiskFailures, a remediation is already in progress on this host.
2026-05-26T17:38:21Z In(14) vsandevicemonitord[2099813]: [281712289536]: Diskgroup/disk naa.50025######847 unmounted successfully for evacReason: SmartDiskFailures

  • /var/run/log/vobd.log - vSAN generated alerts indicating - Impending SMART failure, Device evacuation and unmount and Propagated disk group errors.

2025-10-02T05:08:54.611Z In(14) vobd[2097763]:  [vSANCorrelator] 4657######206us: [esx.problem.vob.vsan.lsom.devicewithsmartfailure] vSAN device naa.50000######abc5 smart health status is impending failure. It will be evacuated and unmounted, consider replacing it.

  • /var/run/log/vmkernel.log - Kernel logs confirm - Propagation of unhealthy state to all member disks and Disk group unmount operations completed successfully

2026-05-26T17:27:59.483Z Wa(180) vmkwarning: cpu25:2098607)WARNING: PLOG: PLOGPropagateError:5012: DDP: Propagating unhealthy(noevac) state from original device 52####bb-####-####-####-486######153

2026-05-26T17:27:59.483Z Wa(180) vmkwarning: cpu25:2098607)WARNING: PLOG: PLOGPropagateErrorInt:4920: vSAN device 52####bb-####-####-####-486######153 is unhealthy.

2026-05-26T17:27:59.483Z In(182) vmkernel: cpu86:2099085)LSOM: LSOMLogDiskEvent:8418: Disk Event unhealthyNoEvac for MD 52####bb-####-####-####-486######153 (naa.50025######847:2)

  • /var/run/log/vsanmgmt.log - vSAN management logs confirm - Disk no longer present in PLOG/CMMDS and Device state recorded as DISK_UNHEALTHY_LATENCY_SMART

2026-05-27T09:38:55.825Z In(14) vsand[2100008]: [opID=2100008 VsanLsomHealth::checkDiskState] Got devResState from devsTelemetry for 52####bb-####-####-####-486######153: DISK_UNHEALTHY_LATENCY_SMART

2026-05-27T09:38:55.826Z Wa(12) vsand[2100008]: [opID=2100008 VsanHealthSystemImpl::_QueryPhysicalDiskHealthSummary] Failed to get disk 52####bb-####-####-####-486######153 rebalance result : (vim.fault.VsanFault) {
2026-05-27T09:38:55.826Z Wa(12)[+] vsand[2100008]:   faultMessage = (vmodl.LocalizableMessage) [
2026-05-27T09:38:55.826Z Wa(12)[+] vsand[2100008]:     (vmodl.LocalizableMessage) {
2026-05-27T09:38:55.826Z Wa(12)[+] vsand[2100008]:       key = 'com.vmware.vsan.health.msg.invaliddiskuuid.error',
2026-05-27T09:38:55.826Z Wa(12)[+] vsand[2100008]:       message = 'Got invalid disk uuid for vSAN rebalance status'
2026-05-27T09:38:55.826Z Wa(12)[+] vsand[2100008]:     }
2026-05-27T09:38:55.826Z Wa(12)[+] vsand[2100008]:   ]
2026-05-27T09:38:55.826Z Wa(12)[+] vsand[2100008]: }

Resolution

  • Replace the failing disk after taking host into maintenance mode (safest approach)

Note : Ensure data evacuation has completed before removing the drive physically.

  • After replacement, add the new disk to the vSAN disk group to restore capacity and redundancy.

If deduplication and compression are enabled, 

  • Take host into maintenance mode.
  • Remove the disk group with failing disk
  • Replace the disk.
  • Recreate the disk group
  • Exit host from maintenance mode
  • Allow vSAN to resync.
Powered by Wolken Software