The ESXi host fails to access the NAS datastore when the connection to NAS datastore becomes APD state and reports the error: Could not allocate xxxx bytes for dynamic heap NetPktHeapLow. Request returned Out of memory

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• The ESXi host cannot access the NAS datastore when the connection to NAS datastore becomes APD state
• The ESXi host fails to access the NAS datastore when the connection to NAS datastore becomes APD state
• In the vmkernel.log file, you see entries similar to:
  
  Could not allocate xxxx bytes for dynamic heap NetPktHeapLow. Request returned Out of memory
  
  
  Device or filesystem with identifier [xxxxxxxx-xxxxxxxx] has entered the All Paths Down state.
  
  
• Increasing the size of network packet buffer pool does not resolve the issue.
  

  
  

  Note: For more information about increasing the network packet buffer pool, see Out of memory with enabled NetQueue when using jumbo frames on ESXi 5.1 and later (2042874).

Environment

VMware vSphere ESXi 5.1

Cause

This issue occurs due to lack of Low Memory in NUMA node. When virtual machines with a small usage of memory are running on ESXi host, virtual machines are balanced between each NUMA nodes.

However, when a virtual machine using large amount of memory runs on a NUMA node (such as NUMA node 0), it uses up the Low Memory mapped to NUMA node. In this case, VMkernel fails to get network heap from Low Memory space for the connection to the NAS datastore. Thus, the connection goes into APD state.

Resolution

To resolve this issue, disable NUMA in BIOS.

For information about how to disable NUMA in BIOS, contact the hardware vendor.

Additional Information

Out of heap memory in ESXi 5.0 with NetQueue and Jumbo Frames enabled
ESXi から NAS データストアへの接続が APD 状態となり、データストアへのアクセスができず次のエラーが出力される： Could not allocate xxxx bytes for dynamic heap NetPktHeapLow. Request returned Out of memory