DFW Memory Usage Very High Due to `vsip-kentries` heap consuming excessive memory.

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

Customers also see vMotion fails with DFW vMotion Failure Alarm - "The DFW vMotion for DFW filter nic-##########-eth0-vmware-sfw.2 on destination host **** has failed and the port for the entity has been disconnected."  as well as observing repeated alerts indicating high memory usage in the vsip-kentries heap, particularly during or after large-scale vMotion operations. This impacts is associated with a large number of address sets being marked as LOCAL instead of GLOBAL after vmotion and filter imports, resulting in memory bloat and failure to release heap post-import.

Environment

NSX 4.2.x

vDefend Firewall

Cause

When a host enters maintenance mode and multiple VMs are vMotioned off in bulk, each vNIC on these VMs triggers an import of its firewall filter. These filters contain address sets which, during import, are temporarily created as LOCAL addrsets.

Normally, after import, the NSX control plane (via cfgAgent) sets the GLOBAL_TABLES flag on the kernel interface (kif), converting these LOCAL addrsets into GLOBAL addrsets—ensuring efficient memory reuse. In this case, due to the high rate of filter imports (e.g., ~100 filters each with ~100 addrsets), the GLOBAL_TABLES flag could not be set in time. Consequently, LOCAL addrsets remained in memory and caused the vsip-kentries heap to cross critical thresholds.

Sample Logs:

Sample output showing high memory usage for vsip-kentries heap:

14 vsip-kentries True 90 91 2813 MB 3070 MB 91 17:32:34

Heap zone details (inUse is significantly high):

zone 8: pfrkentry maxObj = -1, objSize = 176, alloc = 702946228, free = 686395164, inUse = 16551064, numFail = 39099, totalMem = 2912987264

Filter import appears successful in the logs:

2025-06-23T15:14:31.906Z In(182) vmkernel: cpu52:2098112)Importing succeeded

2025-06-23T15:14:31.906Z In(182) vmkernel: cpu52:2098112)Filter creation report: filter = nic-27027322-eth0-vmware-sfw.2, source = Import

However, GLOBAL_TABLES flag is not set on the imported filter:

getkifflags -f nic-27027322-eth0-vmware-sfw.2

kif flags: (0xe40)

PF_KIF_FLAG_GLOBAL_TABLES 0 <<< not set

LOCAL flag still present in addrset:

getaddrsets -f nic-27027322-eth0-vmware-sfw.2 -o

addrset 004148be-3e1f-4f15-b0d8-097a1f40a0e2 {

# generation number: 0

# realization time : 2025-04-29T08:20:55

# refs: 1, 0 flags: 0x10000025 (ROOT,LOCAL,PER,ACT,ANCREF) }

Large number of imports observed in a short span:

2025-06-23T15:13:16.067Z In(182) vmkernel: cpu52:2098112)Importing nic-27025424-eth5647-vmware-sfw.1, Version 1000

...
2025-06-23T15:13:18.671Z In(182) vmkernel: cpu74:2098112)Importing nic-27025424-eth0-vmware-sfw.2, Version 1100

Resolution

A permanent fix is being targeted for a future NSX release 9.0.x and above.

Workarounds (Until Fix is Available)

To mitigate the issue in existing environments:

1. Perform vMotion in smaller batches to avoid overwhelming the filter import process.

2. Use CIDR blocks or dynamic criteria in NSGroups rather than listing individual IP addresses.

3. Eliminate overlapping or excessively large dynamic groups to reduce addrset footprint.

4. If Layer 2 rules aren't in use, modify global_macset_optimization_mode_enabled to be true, which will eliminate mac addresses from address sets and will free up heap space.
   
   1. Get current settings via GET policy/api/v1/infra/settings/firewall/security
   2. Modify global_macset_optimization_mode_enabled to be true instead of false.
   3. Apply the setting via PATCH policy/api/v1/infra/settings/firewall/security