Out of memory with enabled NetQueue when using jumbo frames on ESXi 5.1 and later

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms

• ESXi 5.1 or later hosts with a large number of 32-bit or 64-bit DMA physical NICs and NetQueue enabled, run out of memory when using jumbo frames (MTU is 9000 bytes).
• The lack of memory for network packets leads to lost virtual machine connectivity and may also lose connection with the vCenter Server.
• Network performance of network card is substantially degraded.
• NFS datastores mounted and accessed through this card become unmounted or flap between connected and disconnected state.
• vMotions time out.
• Restarting host management agents fail to complete when they attempt to re-initialize.
• In the vmkernel.log file, located at /var/log/, you see the message:
  
  Could not allocate xxxx bytes for dynamic heap NetPktHeapLow. Request returned Out of memory

Notes:

• For information about configuring the packet memory allocation for DMA-constrained NICs on ESXi 5.1, see Virtual machines running on ESXi 5.1 lose connectivity and network traffic stops if a large number of physical NIC e1000e ports are configured with a jumbo MTU setting of 9000 Bytes (2033170).
• To ensure that you are running a supported configuration, see vSphere 5.1 Configuration Maximums guide.
• For information about the maximum number of physical NICs that a host with enabled jumbo frames can have, see Configuration maximums for NIC ports on ESXi/ESX 4.x and ESXi 5.x (1020808).
• For information on out of heap memory in ESXi 5.0, see Out of heap memory in ESXi 5.0 with NetQueue and Jumbo Frames enabled (2042587).
• For out of heap information related to ESX/ESXi 4.x, see Failed vMotions and degraded network performance with Emulex OneConnect 10GbE adapter and be2net driver (2013966).

Environment

VMware vCenter Server 5.5.x
VMware vSphere CLI 5.5
VMware vSphere PowerCLI 5.0.1
VMware vSphere PowerCLI 5.5
VMware vSphere CLI 5.0
VMware vSphere PowerCLI 5.0
VMware vSphere ESXi 5.1
VMware vCenter Server 5.1.x
VMware vSphere ESXi 5.5

Resolution

To resolve the out-of-memory issue, increase the size of the network packet buffer pool.

ESXi 5.1 uses a buffer pool for network packets. By default, the maximum memory allocated for packets is 656 MB.

ESXi 5.5 uses a buffer pool that is calculated on per-GB basis, so the maximum memory allocated for packets is relative to the amount of physical memory in the system.

To increase the maximum size of the network packet buffer pool and heap on the host, you can use the vSphere Web Client, vSphere Client, vCLI, or PowerCLI commands.

Note: A Reboot is required on the ESXi host for changes to take in effect.

Modifying the net packet pool size from the vSphere Web Client

To modify the net packet pool size from the vSphere Web Client:

1. In the vSphere Web Client, navigate to the host in the vSphere inventory.
2. Click the Manage tab.
3. Select Advanced Settings under System.
4. Change the advanced parameter applicable to your ESXi version.

   ESXi version	Procedure
   5.1	Set the VMkernel.Boot.netPktPoolMaxSize parameter to the target maximum pool size in MB, for example, to 800MB. Set the VMkernel.Boot.netPktHeapMaxSize parameter from 64 to 256 size in MB.
   5.5 and later	Set the VMkernel.Boot.netPktHeapMaxMBPerGB parameter to the target maximum pool size in MB, for example, to 512MB. Set the VMkernel.Boot.netPktPoolMaxMBPerGB parameter from 75 to 200 size in MB.

5. Click OK.
6. Reboot the ESXi host.

Modifying the net packet pool size from the vSphere Client

To modify the net packet pool size from the vSphere Client:

 

1. In the vSphere Client, navigate to the host in the vSphere inventory.
2. Click the Configuration Tab.
3. Select Advanced Settings under Software.
4. Scroll Down and expand VMkernel.
5. Select Boot.
6. Change the advanced parameter applicable to your ESXi version.

   ESXi version	Procedure
   5.1	Set the VMkernel.Boot.netPktPoolMaxSize parameter to the target maximum pool size in MB, for example, to 800MB. Set the VMkernel.Boot.netPktHeapMaxSize parameter from 64 to 256 size in MB.
   5.5 and later	Set the VMkernel.Boot.netPktHeapMaxMBPerGB parameter to the target maximum pool size in MB, for example, to 512MB. Set the VMkernel.Boot.netPktPoolMaxMBPerGB parameter from 75 to 200 size in MB.

7. Click OK.
8. Reboot the ESXi host.

Modifying the network packet heap and pool size on the ESXi host using vCLI commands

To increase the size of the network packet buffer pool, set the netPktPoolMaxSize and netPktHeapMaxMBPerGB parameters by using the esxcli system settings kernel set command on the host:

To set the advanced parameter applicable to your ESXi version:

1. Run the commands applicable to your ESXi version:
    

   ESXi version	Command
   5.1	esxcli system settings kernel set -s netPktHeapMaxSize -v 256 esxcli system settings kernel set -s netPktPoolMaxSize -v 800
   5.5 and later	esxcli system settings kernel set -s netPktHeapMaxMBPerGB -v 512 esxcli system settings kernel set -s netPktPoolMaxMBPerGB -v 200

   
    
2. Run the reboot command to reboot the ESXi host or reboot from the UI.
3. Run these commands and verify that the settings are persistent after reboot:
    

   ESXi version	Command
   5.1	esxcli system settings kernel list | grep netPktHeapMaxSize esxcli system settings kernel list | grep netPktPoolMaxSize
   5.5 and later	esxcli system settings kernel list | grep netPktHeapMaxMBPerGB esxcli system settings kernel list | grep netPktPoolMaxMBPerGB

Modifying the network packet heap and pool size using vSphere PowerCLI

To set the network packet heap and pool size:

1. From a PowerCLI shell, run the commands applicable to your ESXi version:
   
   Connect-VIServer [IP_or_FQDN_of_Host]

   ESXi version	Command
   5.1	Set-VMHostAdvancedConfiguration -VMHost IP_or_FQDN_of_Host -Name VMkernel.Boot.netPktHeapMaxSize -Value 256 Set-VMHostAdvancedConfiguration -VMHost IP_or_FQDN_of_Host -Name VMkernel.Boot.netPktPoolMaxSize -Value 800
   5.5 and later	Set-VMHostAdvancedConfiguration -VMHost IP_or_FQDN_of_Host -Name VMkernel.Boot.netPktHeapMaxMBPerGB -Value 512 Set-VMHostAdvancedConfiguration -VMHost IP_or_FQDN_of_Host -Name VMkernel.Boot.netPktPoolMaxMBPerGB -Value 200

   
    
2. Run the reboot command to reboot the ESXi host or reboot from the UI.
3. Run the commands applicable to your ESXi version to verify that the settings are persistent after reboot:
    

   ESXi version	Command
   5.1	Get-VMHostAdvancedConfiguration -VMHost IP_or_FQDN_of_Host -Name VMkernel.Boot.netPktHeapMaxSize Get-VMHostAdvancedConfiguration -VMHost IP_or_FQDN_of_Host -Name VMkernel.Boot.netPktPoolMaxSize
   5.5 and later	Get-VMHostAdvancedConfiguration -VMHost IP_or_FQDN_of_Host -Name VMkernel.Boot.netPktHeapMaxMBPerGB Get-VMHostAdvancedConfiguration -VMHost IP_or_FQDN_of_Host -Name VMkernel.Boot.netPktPoolMaxMBPerGB

For more information, see the VMware PowerCLI Documentation.