ESXi hosts in All Paths Down (APD) condition may appear as Not Responding in vCenter Server
Article ID: 318938
Updated On:
Products
Issue/Introduction
An ESXi host with one or more LUNs in an All Paths Down (APD) condition may become not manageable in vCenter server and may experience these symptoms:
The ESXi host appears as Disconnected or Not Responding in the vCenter Server inventory
- The Management IP of the ESXi node becomes intermittently unreachable while the physical hardware remains operational.
Virtual machines using the LUNs experiencing the APD may become unresponsive
- From ESXi,
var/log/vobd.logthere is an event for "Device has entered APD state."
YYYY-MM-DDTHH:MM:SS.255Z: [APDCorrelator] 15992267920301us: [esx.problem.storage.apd.start] Device or filesystem with identifier [########-########] has entered the All Paths Down state.YYYY-MM-DDTHH:MM:SS.764Z: [APDCorrelator] 15992274428660us: [esx.clear.storage.apd.exit] Device or filesystem with identifier [########-########] has exited the All Paths Down state.
Where: ########-######## represents the identifier of the datastore. In this example, the identifier is for an NFS datastore, and the details of the datastore can be found in the output of the "esxcfg-info | less" command by searching for that identifier.
Environment
VMware ESXi 7.x
VMware ESXi 8.x
Cause
If datastores that are needed by an ESXi host are no longer accessible because all data paths between the ESXi host and the datastore are no longer passing packets, then the ESXi host may appear as "Not Responding" when viewed from vCenter server.
If the storage used is iSCSI, A common cause of persistent or intermittent APD conditions is a misconfigured or non-standardized iSCSI network topology. If iSCSI storage traffic is not isolated to a dedicated Layer 2 path and is instead routed through a default gateway or firewall, it is subjected to routing delays, packet inspection, and state timeouts. Because iSCSI is highly sensitive to latency, these network delays trigger APD events, causing the host daemon (hostd) to become unresponsive.
Resolution
- Standardize iSCSI Network Topology: Ensure the storage network configuration is uniform across all hosts in the cluster. Work with your network/storage teams to ensure connection to the storage arrays is intact over the iSCSI network.
- Verify Hardware and Fabric: Check Fibre Channel cables and SFP modules for hardware faults. Ensure correct WWN zoning is configured on the fabric switches.
- Restart Management Agents: If SSH is available, attempt to restart management agents to regain connection to vCenter:
/etc/init.d/hostd restart/etc/init.d/vpxa restart
- Hard Reboot: If agents are locked and the CLI is unresponsive, perform a hard reboot of the ESXi host.
Note: A hard reboot is required to clear kernel worker threads that are hung on blocked storage I/O. - Perform a Storage Rescan: Once paths and network topology are corrected, perform a storage rescan to ensure all paths are restored:
esxcli storage core adapter rescan --all
Additional Information
Feedback
