NSX Kernel heap memory used to store DFW rules and containers is exhausted

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• Kernel heap memory used to store DFW rules and containers is exhausted in some scale-out environments where 1000 numbers of SG (Security Groups) and 1000s of IP sets are used
• Distributed firewall rules are reported in the User Interface (UI) as out of sync
• New DFW rules fail to be published
  
  Note: For additional symptoms and log entries, see the Additional Information section.
  

Environment

VMware NSX for vSphere 6.2.x

Resolution

This is a known behavior in VMware NSX for vSphere 6.2.x where heap memory is limited to 1.5GB.

Notes:

• In VMware NSX for vSphere 6.2.4 and later, the heap memory has been increased from 1.5GB to 3GB for Servers with 128GB or more memory. This improves consolidation ratio.
• The increase from 1.5GB to 3GB in NSX for vSphere 6.2.4 is only applicable for vSphere 6.0 and later. If you run NSX for vSphere 6.2.4 on vSphere 5.5, the 1.5GB limit still applies.
• In VMware NSX for vSphere 6.3.0, Global Address Sets is introduced which significantly reduces heap memory. Global Address Sets are disabled by default.

To avoid this issue, VMware recommends you to use one of these options:

• Ensure that free heap memory is always greater than 20%.
  

• Configure Distributed Resource Scheduling (DRS) to Manual mode and migrate virtual machines manually to maintain the right consolidation ratio.
  

Note: For more information on design recommendations, see the Professional Services.

Additional Information

You experience these additional symptoms:

• In the /var/log/vmkwarning.log file on the ESXi host, you see entries similar to:
  
  WARNING: Heap: 3583: Heap vsip already at its maximum size. Cannot expand.

• In the /var/log/vsfwd.log file on the ESXi host, you see entries similar to:
  
  INFO messagingTaskExecutor-2 SystemEventDaoImpl:133 - [SystemEvent] Time:'Wed Jun 08 14:38:00.764 AEST 2016', Severity:'Major', Event Source:'host-10696', Code:'301032', Event Message:'Failed to apply firewall rule to vnic.', Module:'vShield Firewall', Universal Object:'false'
  INFO messagingTaskExecutor-2 ConfigurationPublisher:229 - Updating host host-10696 status for firewall, generation -1 ; StatusCode - 301032, Status Message - eventId: 301032 -- The Status Code "StatusCode - 301032" tell that the vibs is failing to install on the respective host which is "host-10696"(company.com).
  vsfwd: [WARN] failed to apply ruleset: agent vmware-sfw vnic 500ab00d-ce7b-ca98-2016-5c297dd6e703.000 gennum 1464955567691
  vsfwd: [WARN] failed to execute command CreateAddrSet: out of memory
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
  

DFW ルールとコンテナの格納に使用する NSX カーネル ヒープ メモリが枯渇する
用于存储 DFW 规则和容器的 NSX 内核堆内存耗尽