Halting PDL messages on ESXi when a LUN is unexpectedly removed at array level

Products

• VMware vSphere ESXi

Issue/Introduction

Excerpts similar to below are seen In /var/log/vmkernel.log:

YYYY-MM-DD:HH:MM:SS  cpu#:#######)WARNING: NMP: nmp_PathDetermineFailure:1234: Cmd (0x1a) PDL error (0x5/0x25/0x0) - path vmhbaX:C0:T1:L0 device naa.############################## - triggering path failover
YYYY-MM-DD:HH:MM:SS  cpu#:#######)WARNING: NMP: nmp_PathDetermineFailure:1234: Cmd (0x1a) PDL error (0x5/0x25/0x0) - path vmhbaX:C0:T1:L0 device naa.############################## - triggering path failover
YYYY-MM-DD:HH:MM:SS  cpu#:#######)WARNING: NMP: nmp_PathDetermineFailure:1234: Cmd (0x1a) PDL error (0x5/0x25/0x0) - path vmhbaX:C0:T4:L0 device naa.##############################- triggering path failover
YYYY-MM-DD:HH:MM:SS  cpu#:#######)WARNING: NMP: nmp_PathDetermineFailure:1234: Cmd (0x1a) PDL error (0x5/0x25/0x0) - path vmhbaX:C0:T4:L0 device naa.############################## - triggering path failover
YYYY-MM-DD:HH:MM:SS  cpu#:#######)WARNING: NMP: nmp_PathDetermineFailure:1234: Cmd (0x1a) PDL error (0x5/0x25/0x0) - path vmhbaX:C0:T5:L0 device naa.############################## - triggering path failover
YYYY-MM-DD:HH:MM:SS  cpu#:#######)WARNING: NMP: nmp_PathDetermineFailure:1234: Cmd (0x1a) PDL error (0x5/0x25/0x0) - path vmhbaX:C0:T5:L0 device naa.##############################- triggering path failover

Running the commands on ESXi's SSH:

esxcli storage vmfs extent list 

• The device is present within the list of visible extents 
• The size is shows as 0 bytes

 esxcli storage vmfs filesystem list

• No datastore is found to be associated with the LUN.
• No RDM's in use with the listed device. 

Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

 

Cause

When a LUN is unpresented from ESXi without the device being 'detached' from the host, the host logs might continually spew log excerpts regarding Permanent device loss (PDL). 

Resolution

• Log in to vCenter Server and for each relevant host:
  • Navigate to its storage device configuration.
  • Identify the unpresented LUN(s) (e.g. by their NAA ID).
  • If currently attached, perform the detach operation.
  • Once completed for all hosts, initiate a storage rescan on those hosts

Read more about Permanent Device Loss (PDL) and All-Paths-Down (APD) on host here.