All Paths Down (APD) following mass vMotion triggered by DRS Mandatory Affinity Rule

Products

• VMware vCenter Server

Issue/Introduction

• ESX host reports an All Paths Down (APD) warning for NFS datastores

• Host's /var/log/vmkernel.log shows log entries similar to:
  vmkernel: cpu##:#######)StorageApdHandlerEv: ###: Device or filesystem with identifier [########-########] has entered the All Paths Down state.

• The /var/log/vmkernel.log displays warnings regarding low memory states during migrations
  WARNING: VMotionRecv: ####: #### D: World #### waited #### seconds to leave low memory state on dest host during pre-copy. May cause unexpected VMotion failures!

• The /var/log/vmkernel.log may show "failed memory allocation" messages for the specific network card driver in use (e.g. in this example the Cisco enic driver is affected):
  vmkwarning: cpu##:#######)WARNING: nenic: enic_rq_alloc_buf:###: [####:##:##.#] enic_rq_alloc_buf:vmk_PktAlloc failed memory allocation, status:bad0014
Wa(180) vmkwarning: cpu##:#######)WARNING: Heap: ####: Could not allocate 8192 bytes for dynamic heap NetPktHeapLow. Request returned Out of memory

• Virtual machine and application performance degrades or becomes unresponsive during the event.

Environment

• VMware vCenter Server 8.x
• VMware vCenter Server 9.x
• VMware ESX 8.x
• VMware ESX 9.x
• NFS Storage Servers (i.e. NetApp)

Cause

The implementation of a Mandatory DRS affinity rule can force the simultaneous migration of multiple large virtual machines to a single target host. This concurrent migration activity causes the destination host to enter a critical low memory state. The resulting resource contention impacts the host's ability to maintain network heartbeat and I/O processing for NFS storage, triggering the All Paths Down (APD) event.

Resolution

1. Disable or Modify the DRS Rule:

   1. Log in to the vSphere Client.

   2. Navigate to the Cluster → Configure → Configuration → VM/Host Rules.

   3. Locate the rule (e.g., Application Servers) and click Edit.

   4. Either uncheck Enable to suspend the rule or change the type from Must run on hosts in group to Should run on hosts in group (Soft Affinity). This allows DRS to prioritize host health over strict rule compliance during resource contention.

2. Manual Resource Rebalancing:

   1. If the VMs must reside on a specific host, migrate them manually one at a time via vMotion to monitor host memory impact.

   2. Ensure the destination host has sufficient unreserved memory to accommodate the full memory footprint of all incoming VMs before enabling the rule.

3. Validate Host Health:

   1. Once migrations are complete or the rule is disabled, verify that the storage paths have recovered from the APD state and that the host is no longer reporting a low memory condition.

Additional Information

Using VM-Host Affinity Rules

Permanent Device Loss (PDL) and All-Paths-Down (APD) on host