vSAN ESA NVMe Controller PDL Alerts with Healthy Hardware Status (VMware vSAN)

Products

• VMware vSAN

Issue/Introduction

This article addresses scenarios where VMware vSAN reports a Permanent Device Loss (PDL) or "Unhealthy" disk alert, but physical hardware management consoles (such as iDRAC, xClarity, or iLO) report the drive as healthy. This discrepancy typically indicates a controller-level I/O failure on the NVMe device rather than a physical removal or connectivity issue.

Symptoms:

• vCenter displays: "One of the disks is detected with PDL" or "Disk Group Unhealthy."
• Hardware management console shows physical drive/slot as healthy.
• esxcli vsan storage list shows the device as missing or "in CMMDS: false."
• Logs (vobd/vmkwarning) indicate esx.problem.vob.storage.connectivity.lost or NVMEDEV: Controller failed to recover.

Environment

• VMware vSAN 8.x / 9.x (ESA)
• All supported server hardware (Dell, HPE, Lenovo)

Cause

The hardware console performs out-of-band electrical presence checks, which only verify if the drive is physically seated in the PCIe slot. ESXi checks for active storage path responsiveness under I/O load. If the NVMe controller on the drive fails to process I/O requests, ESXi will mark the device as PDL even if the drive is technically "powered on" and visible to the server BIOS/iDRAC.

Resolution

1. Verify the drive status in the ESXi command line: # esxcli vsan storage list.
2. If the device is marked as PDL, review vmkwarning logs for controller recovery failures (e.g., "Controller failed to recover after X attempts").
3. Perform a host reboot if production load permits, to rule out a transient PCIe state.
4. If the alert persists after a reboot, the NVMe controller is non-operational. Open a case with the hardware vendor to dispatch a replacement drive.
5. Physically replace the failed NVMe drive in the identified slot.
6. Once the physical drive is replaced, log in to the vSphere Client.
7. Navigate to Cluster > Configure > vSAN > Disk Management.
8. Claim the new drive into the vSAN storage pool.