iSCSI Datastore PDL After metro-cluster Storage Failover

Products

• VMware vSphere ESXi

Issue/Introduction

• iSCSI datastores enter a Permanent Device Loss (PDL) or All Paths Down (APD) state during a storage failover event (e.g., metro-cluster failover).
• The storage arrays and/or network infrastructure experienced concurrent reboots alongside the ESXi host.

Environment

VMware vSphere ESXi (All versions)

Cause

This issue is caused by a timing discrepancy during the initialization of the environment's "infrastructure layers." When ESXi hosts and the underlying storage/network fabric are powered on or rebooted simultaneously, the ESXi software iSCSI discovery process may attempt to connect to targets before the storage array controllers or network switches have fully initialized. Because the storage layer is not yet ready to respond, the iSCSI initiator fails to register redundant paths during the boot-time discovery phase.

As a direct result of this boot-time failure, the ESXi host shows only partial path registration prior to any failover, displaying, for example, only 2 paths registered instead when the expected is 4. If this discrepancy is not caught and a storage failover occurs later, the host has no redundant paths available to maintain connectivity, ultimately resulting in a Permanent Device Loss (PDL).

Entries in /var/run/log/vmkernel.log shows iSCSI login failures or discovery timeouts occurring only during the initial host boot sequence: iscsid: connection to discovery address failed iscsid: connection login retries exceeded

Additionally, on /var/run/log/vmkernel.log, "Tcpip: 3472: msleep returned 4" error messages are seen during the iscsi discovery on host boot sequence.

Resolution

This is a procedural and sequencing issue rather than a software defect. To ensure full path redundancy:

1. Verify Boot Sequencing: Ensure that storage arrays and network fabric are fully operational and reachable before powering on ESXi hosts.
2. Post-Boot Validation: After any site-wide power event or simultaneous reboot, administrative teams must verify the path count on all ESXi hosts.
   • Run: esxcli storage core path list | grep "Device:" | sort | uniq -c
   • Confirm the count matches the multipathing design.