Permanent disk failure in vSAN
search cancel

Permanent disk failure in vSAN

book

Article ID: 391710

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

A vSAN disk enters a permanent error state when cumulative I/O retry timeouts or SMART read errors exceed internal monitoring thresholds. This article provides steps to verify the failure and initiate replacement.

 

Symptoms:

  • In vCenter UI, the Skyline health (vSAN cluster > Monitor > vSAN > Skyline Health > Physical Disk > Operation health) may report permanent disk failure for the vSAN disk as in the figure below:

  • In "/var/log/vobd.log" of ESXi host will also report the disk is under permanent error:
YYYY-MM-DDTHH:MM:SSZ:[vSANCorrelator] 12219239351514u3:[esx.problem.vob.vsan.pdl.offline] vSAN device ########-####-####-####-########### has gone offline.
YYYY-MM-DDTHH:MM:SSZ:[vSANCorrelator] 12219239351514u3:[esx.problem.vob.vsan.lsom.devicerepair] Device e########-####-####-####-########### is in offline state and is getting repaired.
YYYY-MM-DDTHH:MM:SSZ:[vSANCorrelator] 12219239351514u3:[vob.vsan.pdl.offline] vSAN device e########-####-####-####-########### has gone offline.
YYYY-MM-DDTHH:MM:SSZ:[vSANCorrelator] 12219239351514u3:[esx.problem. vob.vsan.pdl.offline] vSAN device e########-####-####-####-########### has gone offline.
YYYY-MM-DDTHH:MM:SSZ:[vSANCorrelator] 12219239351514u3:[esx.problem.vob.vsan.lsom.diskerror] vSAN device #### is under permanent error.
   
                                                                          (or)

YYYY-MM-DDTHH:MM:SSZ In(14) vobd[2098148]:  [vSANCorrelator] 9883129325851us: [vob.vsan.pdl.offline] vSAN device 5####3c-#####5##d-d####3-######1 has gone offline.
YYYY-MM-DDTHH:MM:SSZ In(14) vobd[2098148]:  [APDCorrelator] 9882990619157us: [esx.problem.storage.apd.start] Device or filesystem with identifier [##########] has entered the All Paths Down state.
YYYY-MM-DDTHH:MM:SSZ In(14) vobd[2098148]:  [vSANCorrelator] 9882990619196us: [esx.problem.vob.vsan.pdl.offline] vSAN device 5####c-1626-####-####-f01#####1 has gone offline.
YYYY-MM-DDTHH:MM:SSZ In(14) vobd[2098148]:  [psastorCorrelator] 9882990619971us: [esx.problem.storage.connectivity.lost] Lost connectivity to storage device #############. Path vmhba0:C0:T0:L0 is down. Affected datastores: Unknown.
YYYY-MM-DDTHH:MM:SSZ In(14) vobd[2098148]:  [psastorCorrelator] 9883129325791us: [vob.psastor.device.state.permanentloss] Device :eui.############ has been removed or is permanently inaccessible.
YYYY-MM-DDTHH:MM:SSZ In(14) vobd[2098148]:  [psastorCorrelator] 9882990620278us: [esx.problem.psastor.device.state.permanentloss] Device: eui.############### has been removed or is permanently inaccessible. Affected datastores (if any): Unknown.

  • Physical disk failure can be confirmed by logging in to hardware interface.

    Example from iLO:



  • Incase the vSAN disks are identified as unhealthy and they are going to face failure eventually. In var/run/log/vobd.log, you will see below entries -

    YYYY-MM-DDTHH:MM:SSZ In(14) vobd[2097812]:  [vSANCorrelator] 690804744us: [esx.problem.vob.vsan.lsom.diskunhealthy] vSAN device 528c1c3d-ddd2-7210-c721-############ is unhealthy.
    YYYY-MM-DDTHH:MM:SSZ In(14) vobd[2097812]:  [vSANCorrelator] 690811954us: [esx.problem.vob.vsan.lsom.diskunhealthy] vSAN device 5207b712-e8a7-2b82-3283-############ is unhealthy.

  • ESXi host localcli storage core device smart get -d <device> shows a high Read Error Count (e.g., 1,320,182).
  • Host /var/log/vsandevicemonitord.log indicates the disk resilience state: Device <naa.ID> state is DISK_UNDER_PERM_ERROR
  • To check the Impending failure state of the unhealthy disks.

[root@esxi01:~] localcli storage core device smart get -d naa.###################
Parameter                 Value              Threshold  Worst  Raw
------------------------  -----------------  ---------  -----  ---
Health Status             IMPENDING FAILURE  N/A        N/A    N/A
Media Wearout Indicator   0                  100        N/A    N/A
Write Error Count         0                  N/A        N/A    N/A
Read Error Count          0                  N/A        N/A    N/A
Power Cycle Count         0                  N/A        N/A    N/A
Reallocated Sector Count  0                  N/A        N/A    N/A
Drive Temperature         27                 N/A        N/A    N/A
Write Sectors TOT Count   5239103362346      N/A        N/A    N/A
Read Sectors TOT Count    2353051044834      N/A        N/A    N/A
Program Fail Count        0                  N/A        N/A    N/A
Erase Fail Count          0                  N/A        N/A    N/A

Environment

  • VMware Cloud Foundation (VCF)
  • VMware vSAN 8.x (OSA and ESA)

Cause

The vsandevicemonitord service transitions a disk to DISK_UNDER_PERM_ERROR when hardware-level I/O failures exceed threshold limits, necessitating replacement to maintain cluster redundancy

Resolution

  1. Identify the failing disk UUID via vCenter UI or esxcli vsan storage list.
  2. Validate storage controller driver and firmware versions against the VMware Compatibility Guide.
  3. Check the SMART health status using localcli storage core device smart get -d <naa.id>.
  4. Engage the hardware vendor to perform physical diagnostics and confirm media failure.
  5. Replace the defective hardware using the procedure for your specific architecture:
  6. Verify the new disk is added to the disk group and monitor the resync progress.
  7. To speak with a customer representative or a Support Engineer see Contact Support (Link: https://support.broadcom.com/web/ecx/contact-support). Scroll to the bottom of the page and click on your respective region.

Additional Information

vSANでの永続的なディスク障害

Powered by Wolken Software