ESXi host lost access to datastore

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms: 

• All virtual machines associated with the affected datastore were showing as invalid.
• vCenter may be down if it resides on the affected datastore.

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

The loss of connectivity to the storage device is causing the datastore to be inaccessible. The error "vmhba##:C#:T#:L# (PERM LOSS)" indicates that the ESXi host is unable to communicate with the device at the specified path due to a permanent loss of connection. This can  be caused due to a problem with the storage device or  LUN. 

• In the /var/run/log/vmkernel.log file, you see the message:

vmkernel: cpu#:2097648)VMW_SATP_ALUA: satp_alua_issueCommandOnPath:1019: Path "vmhba##:C#:T#:L#" (PERM LOSS) command 0xa3 failed with status Device is permanently unavailable. H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x25 0x0.

• In the /var/run/log/hostd.log file, you see the message:

Hostd[2099261]: -->          value = "naa.##############################"
Hostd[2100061]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 338 : Lost connectivity to storage device naa.##############################. Path vmhba##:C#:T#:L# is down. Affected datastores: Unknown.

Resolution

It is advised to contact the storage vendor to investigate the PDL

 

Additional Information

For more detail on APD/PDL please refer below KB article.

Permanent Device Loss (PDL) and All-Paths-Down (APD) in vSphere ESXI
Path Redundancy to Storage Device Degraded message on the ESXi host