ESXi hosts in All Paths Down (APD) condition may appear as Not Responding in vCenter Server
search cancel

ESXi hosts in All Paths Down (APD) condition may appear as Not Responding in vCenter Server

book

Article ID: 318938

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Symptoms:

An ESXi host with one or more LUNs in an All Paths Down (APD) condition may become not manageable in vCenter server and may experience these symptoms:

  • The ESXi host appears as Disconnected or Not Responding in the vCenter Server inventory

  • Virtual machines using the LUNs experiencing the APD may become unresponsive

Environment

VMware ESXi 7.x
VMware ESXi 8.x

Cause

If datastores that are needed by an ESXi host are no longer accessible because all data paths between the ESXi host and the datastore are no longer passing packets, then the ESXi host may appear as "Not Responding" when viewed from vCenter server. 

To validate if this is the case, determine whether there are any LUNs (datastores) in an All Paths Down (APD) state on an ESXi host:

From ESXi, var/run/log/vobd.log there is an event for "Device has entered APD state."

YYYY-MM-DDTHH:MM:SS.255Z: [APDCorrelator] 15992267920301us: [esx.problem.storage.apd.start] Device or filesystem with identifier [########-########] has entered the All Paths Down state.
YYYY-MM-DDTHH:MM:SS.764Z: [APDCorrelator] 15992274428660us: [esx.clear.storage.apd.exit] Device or filesystem with identifier [########-########] has exited the All Paths Down state.

Where:  ########-######## represents the identifier of the datastore.  In this example, the identifier is for an NFS datastore, and the details of the datastore can be found in the output of the "esxcfg-info | less" command by searching for that identifier.

Resolution

Follow the below steps to restore host connectivity:

  1. Check Fibre Channel cables and SFP modules for hardware faults.

  2. Ensure correct WWN zoning is configured on the fabric switches.

  3. If SSH is available, attempt to restart management agents: /etc/init.d/hostd restart /etc/init.d/vpxa restart

  4. If agents are locked and the CLI is unresponsive, perform a hard reboot of the ESXi host.
    • Note: A hard reboot is required to clear kernel worker threads that are hung on blocked storage I/O.

  5. Perform a storage rescan to ensure all paths are restored: esxcli storage core adapter rescan --all

If the datastore in question is an NFS datastore (as in the above example), then connectivity can be confirmed by following the steps in Troubleshooting NFS datastore connectivity issues

Additional Information

Powered by Wolken Software