vSAN Skyline Health reports disks in an Absent or Permanent Device Loss (PDL) state. Despite physical replacement of the drives, the failure recurs in the same bays.

Products

• VMware vSAN 8.x

Issue/Introduction

Symptoms: 

• vSAN capacity disks enter PDL(Permanent Device Loss) or APD (All Paths Down) state.
  
  
• Repeated hardware replacements fail to resolve the issue.
• Hardware: HPE ProLiant DL380 Gen11:
  
  
  • Storage: Samsung NVMe SSD Controller PM173Xa
  • Disk Model: VO015360KYDNB
  • Firmware: HPK5

  • Model : NVMe

  • NVME Revision : HPK5

     VID   : 144d    
     DID   : a825  
     SVID  : 1590  
     SDID  : 03ac  

Environment

• VMware ESXi 8.0 Update 3
• VMware vSAN 8.0 Update 3

Cause

Samsung NVMe SSD models running firmware version HPK5 are susceptible to an Uncorrectable Machine Check Exception (UMCE) or an uncorrectable PCIe bus error on AMD and Intel-based Gen11 servers. This condition triggers an NVMe controller failure, resulting in an All Paths Down (APD) or PDL state within the ESXi host.

Refer : HPE Support Advisory

Resolution

Upgrade the SSD firmware to version HPK6 or later to resolve the uncorrectable error condition.

1. Coordinate with the hardware vendor to apply firmware version HPK6 (or later) to all affected SSD models.
2. If disks are in a failed or PDL state, remove the affected devices from the vSAN disk group. 
   
   Refer KB # for more details : KB # 326859 : Troubleshooting vSAN OSA disk issues.
3. To remove a failed disk via command line when the UI option is unavailable.
   
   esxcli vsan storage remove -u <disk_uuid>
4. Recreate the disk group once the firmware update is complete.
   
   Refer : Recreate a Disk Group in vSAN Cluster
5. Monitor vSAN Skyline Health to confirm stable operational status.