Permanent disk failure in vSAN

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• In vCenter UI, the Skyline health (vSAN cluster > Monitor > vSAN > Skyline Health > Physical Disk > Operation health) may report permanent disk failure for the vSAN disk as in the figure below:

 

• /var/log/vobd.log of ESXi host will also report the disk is under permanent error:

025-02-04T15:47:46.212Z:[vSANCorrelator] 12219239351514u3:[esx.problem. vob. vsan.pdl. offline] vSAN device ########-####-####-####-########### has gone offline.
025-02-04T15:47:46.212Z:[vSANCorrelator] 12219239351514u3:[esx.problem. vob. vsan. lsom. devicerepair] Device e ########-####-####-####-########### is in offline state and is getting repaired.
025-02-04T15:47:46.212Z:[vSANCorrelator] 12219239351514u3:[vob.vsan.pdl.offline] vSAN device e ########-####-####-####-########### has gone offline.
025-02-04T15:47:46.212Z:[vSANCorrelator] 12219239351514u3:[esx.problem. vob.vsan.pdl.offline] vSAN device e ########-####-####-####-########### has gone offline.
025-02-04T15:47:46.212Z:[vSANCorrelator] 12219239351514u3:[vob. vsan.lsom. diskerror] vSAN device e ########-####-####-####-########### is under permanent error.

   

                                                                          (or)

• 2025-12-12T23:06:59.125Z In(14) vobd[2098148]:  [vSANCorrelator] 9883129325851us: [vob.vsan.pdl.offline] vSAN device 5####3c-#####5##d-d####3-######1 has gone offline.
2025-12-12T23:06:59.125Z In(14) vobd[2098148]:  [APDCorrelator] 9882990619157us: [esx.problem.storage.apd.start] Device or filesystem with identifier [##########] has entered the All Paths Down state.
2025-12-12T23:06:59.125Z In(14) vobd[2098148]:  [vSANCorrelator] 9882990619196us: [esx.problem.vob.vsan.pdl.offline] vSAN device 5####c-1626-####-####-f01#####1 has gone offline.
2025-12-12T23:06:59.126Z In(14) vobd[2098148]:  [psastorCorrelator] 9882990619971us: [esx.problem.storage.connectivity.lost] Lost connectivity to storage device #############. Path vmhba0:C0:T0:L0 is down. Affected datastores: Unknown.
2025-12-12T23:06:59.126Z In(14) vobd[2098148]:  [psastorCorrelator] 9883129325791us: [vob.psastor.device.state.permanentloss] Device :eui.############ has been removed or is permanently inaccessible.
2025-12-12T23:06:59.126Z In(14) vobd[2098148]:  [psastorCorrelator] 9882990620278us: [esx.problem.psastor.device.state.permanentloss] Device: eui.############### has been removed or is permanently inaccessible. Affected datastores (if any): Unknown.

Environment

VMware vSAN 7.x
VMware vSAN 8.x

Cause

This is caused due to hard disk issue.

vSAN will mark a disk offline when it encounters I/O failure, preventing further operations on the affected disk.

Resolution

If the drive is present and experiencing permanent errors, it may be due to the driver/firmware of the controller as well. Please review the hardware compatibility and apply the appropriate driver/firmware. 

If the drive/s still see the issue, please involve hardware vendor to review the drive health and assist with the next steps.

If its OSA cluster (capacity disk) follow below instructions

Replace a Capacity Device in vSAN OSA Cluster

If its OSA Cluster (Cache disk) 

Replace a Flash Caching Device on a Host in vSAN Cluster

If its ESA cluster follow below instructions to replace the disk 

Replace a Storage Pool Device in vSAN ESA Cluster