vCenter Server alarm "Cannot connect to Storage" and All Paths Down for a storage device

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

An Alarm is triggered in the vCenter Server for multiple hosts "Cannot connect to Storage".

Reviewing the event tab for these hosts, events can be seen with the description:

Device of filesystem with identifier ########-########-####-############ has entered the All Paths Down state.

The ESXi host shows as disconnected or not responding in vCenter Server
Reconnecting the ESXi host using the vSphere Client fails
Reviewing the storage paths in vSphere Client (ESXi→Configure→Storage→Adapters→Paths) shows all of them in dead/error state.

In /var/run/log/vmkernel.log on the ESXi host, there may be entries similar to:

cpu1:2049)WARNING: NMP: nmp_IssueCommandToDevice:2954:I/O could not be issued to device "naa.60##############3" due to Not found
cpu1:2049)WARNING: NMP: nmp_DeviceRetryCommand:133:Device "naa.60##############3": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.
cpu1:2049)WARNING: NMP: nmp_DeviceStartLoop:721:NMP Device "naa.60##############3" is blocked. Not starting I/O from device.
cpu1:2642)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60##############3" - issuing command 0x4124007ba7c0
cpu1:2642)WARNING: NMP: nmpDeviceAttemptFailover:658:Retry world failover device "naa.60##############3" - failed to issue command due to Not found (APD), try again...
cpu1:2642)WARNING: NMP: nmpDeviceAttemptFailover:708:Logical device "naa.60##############3": awaiting fast path state update...

The log of the HA agent, /var/run/log/fdm.log, contains messages like this example:

YYYY-MM-DDThh:mm:ss fdm Db(###) Fdm[########]: [Originator@#### sub=Invt opID=placementService.cpp:###-########] Host host-####### cannot access VM's home datastore: /vmfs/volumes/########-########

Furthermore there might be messages like the one below in /var/run/log/vobd.log on the host:

YYYY-MM-DD T00:26:51.504Z: [APDCorrelator] 2682686563317us: [esx.problem.storage.apd.timeout] Device or filesystem with identifier [########-########] has entered the All Paths Down Timeout state after being in the All Paths Down state for 140 seconds. I/Os will now be fast failed.

Notes:

There might also be messages like the following one in /var/run/log/vmkernel.log:
```
T13:41:33.250z cpu4:8598)StorageApdHandler: 692: APD Handle Created with lock.
```
This log message indicates that the paths to a storage device have been disconnected, and the APD timer has started, but the device has not entered APD state yet.
By default the APD timer runs for 140 seconds, during which the host will continue retrying attempts to re-establish connectivity.
Once the timeout has expired without connectivity being restored on any of the paths, the storage device will enter the actual APD state.
The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on the environment.

Environment

VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 8.0.x
VMware vSphere ESX 9.x

Cause

All-Paths-Down (APD) situation occurs when all paths to a device are down. As there is no indication of whether this is a permanent or temporary device loss, the ESXi host keeps reattempting to establish connectivity. APD-style situations commonly occur when the LUN is incorrectly un-presented from the ESXi host.

The timeout period begins when the storage device becomes unavailable to the ESXi host and enters the APD state. By default, the APD timeout is set to 140 seconds. While the timeout lasts, the host continues its attempts to reestablish connectivity with the device. When the timeout ends and the device does not recover, the host stops its attempts to retry any I/O that is not coming from virtual machines.
The reasons for an APD state can be, for example, a failed switch or a disconnected storage cable.

Resolution

Due to the nature of an APD situation, there is no clean way to recover.

The APD situation needs to be resolved at the storage array/fabric layer to restore connectivity to the host.
All affected ESXi hosts may require a reboot to remove any residual references to the affected devices that are in an APD state.
To resolve this issue, identify the cause of the disconnected LUNs by reviewing the environment, such as Storage array, SAN switch, Device failure, etc.

If the virtual machines on the datastores remain responsive, power off the virtual machines or migrate them to a different datastore or host.

Additional Information

For information on block storage devices, see Path Redundancy to Storage Device Degraded.
For information on NFS storage devices, see Troubleshooting NFS datastore connectivity issues.

A storage device is considered to be in the APD state when it becomes unavailable to the ESXi host for an unspecified period of time. In contrast with the permanent device loss (PDL) state, the host treats the APD state as transient and expects the device to be available again. For more information, see Handling Transient APD Conditions.

japaneseKB:ストレージデバイスのすべてのパスがダウン