Error: Permanent Device Loss (PDL) causing management agent deadlock in ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

• ESXi host appears as "Disconnected" or "Unresponsive" in vCenter Server.
• vpxa and hostd services crash or become unresponsive within seconds of a manual restart.

Environment

VMware vSphere ESXi (All versions)

Cause

This issue occurs when a LUN is unmapped or turned off at the storage array level without first being detached from the ESXi host. ESXi enters a high-overhead retry cycle and polling loop against the unmapped device. Over time, these kernel structure overheads exhaust host resources, causing a deadlock in the management agents (hostd and vpxa).

Log entries in /var/run/log/vmkernel.log show:

• Device naa.#### has been turned off administratively.
• WARNING: VMW_SATP_ALUA: satp_alua_getTargetPortInfo:###: Could not get page 83 INQUIRY data for path "vmhba#:C#:T#:L#" - Device is permanently unavailable.
• H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x25 0x0.
• WARNING: vmw_psp_rr: psp_rrSelectPathToActivate:###: Could not select path for device "naa.####".

Resolution

1. Access the affected host via SSH.
2. Check the status of hostdand vpxa services by running:
   • /etc/init.d/hostd status
   • /etc/init.d/vpxa status
3. If they are not running, restart the services:
   • /etc/init.d/hostd restart
   • /etc/init.d/vpxa restart
4. If management services (hostd/vpxa) fail to stay running after a restart, a host reboot is required to clear deadlocked processes and purge stale PDL kernel structures.
5. To prevent future occurrences, always unmount the datastore and detach the device in ESXi before unmapping the LUN from the storage array. See How to detach a LUN device from ESXi hosts (323128).

Additional Information

Permanent Device Loss (PDL) and All-Paths-Down (APD) on host

ESXi Host Unresponsive with PDL Errors and Cessation of Logging

ESXi host becomes unresponsive following repeated PDL errors on paths