DFW Memory Usage Very High Due to `vsip-kentries` heap consuming excessive memory
search cancel

DFW Memory Usage Very High Due to `vsip-kentries` heap consuming excessive memory

book

Article ID: 404061

calendar_today

Updated On:

Products

VMware vDefend Firewall VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

Customers may observe repeated alerts indicating high memory usage in the vsip-kentries heap, crossing critical thresholds (90%+). Additionally, vMotion operations may fail with the alarm: "The DFW vMotion for DFW filter nic-##########-eth0-vmware-sfw.2 on destination host **** has failed and the port for the entity has been disconnected."

This issue typically occurs during large-scale vMotion operations or during ESXi Live Patching campaigns that utilize Fast Suspend Resume (FSR). The memory bloat is caused by a large number of address sets being permanently marked as LOCAL instead of GLOBAL after a burst of filter imports, resulting in a failure to release the heap post-import.

Note: Rebooting an affected ESXi host is ineffective at clearing this condition. The alarm will typically return shortly after the host boots.

Environment

NSX 4.2.x

vDefend Firewall

Cause

When a host enters maintenance mode and multiple VMs are vMotioned in bulk, or when an ESXi Live Patch is applied causing a Fast Suspend Resume (FSR) on running VMs, each vNIC on these VMs triggers an import of its firewall filter. (Note: FSR acts architecturally as a "local vMotion," forcing a vNIC detach/reattach and a full filter rebuild).

These filters contain address sets which, during import, are temporarily created in the heap as LOCAL addrsets. In environments with large dynamic groups, a single filter can contain tens of thousands of IP and MAC addresses.

Normally, after import, the NSX control plane (via cfgAgent) sets the GLOBAL_TABLES flag on the kernel interface, converting these LOCAL addrsets into GLOBAL addrsets—ensuring efficient memory reuse. However, due to the high rate of concurrent filter imports during bulk vMotions or simultaneous FSRs, the temporary LOCAL tables inflate the heap so rapidly that it hits its maximum limit (e.g., ~3 GB). Once the heap is exhausted (Out of Memory), the cfgAgent lacks the memory overhead required to push the configuration and set the GLOBAL_TABLES flag. Consequently, the addrsets remain permanently stranded in memory as LOCAL.

Rebooting the ESXi host does not resolve the issue. The subsequent VM power-on or DRS repopulation during host recovery triggers a new storm of concurrent filter imports, immediately recreating the race condition and refilling the heap.

 

Sample Logs:

  • In /var/run/log/nsx-syslog you may see a dfw_vmotion_failure message.
2026-01-06T07:45:56.273Z Er(179) cfgAgent[2107114]: NSX 2107114 - [nsx@6876 comp="nsx-controller" subcomp="cfgAgent" s2comp="nsx-monitoring" entId="########-####-####-####-###########" tid="5D785700" level="fatal" eventState="On" eventFeatureName="distributed_firewall" eventSev="critical" eventType="dfw_vmotion_failure"] The DFW vMotion for DFW filter nic-#########-eth0-vmware-sfw.2 on destination host <host.fqdn> has failed and the port for the entity has been disconnected.

 

  • In /var/run/log/vmkernel.log you may see:
2026-01-06T07:45:44.565Z In(182) vmkernel: cpu64:2098147)filter nic-#########-eth0-vmware-sfw.2 flushing flow cache
2026-01-06T07:45:44.565Z In(182) vmkernel: cpu64:2098147)pfr_attach_table: nic-#########-eth0-vmware-sfw.2: ERROR ***************** local root table ########-####-####-####-########### not found
2026-01-06T07:45:44.565Z In(182) vmkernel: cpu64:2098147)pfr_attach_table: nic-#########-eth0-vmware-sfw.2: ERROR ***************** local root table ########-####-####-####-########### not found
2026-01-06T07:45:44.565Z In(182) vmkernel: cpu64:2098147)pfioctl: DIOCADDRULE failed with error 22
2026-01-06T07:45:44.565Z In(182) vmkernel: cpu64:2098147)RAL: pfioctl: error, calling pf_rm_rule
2026-01-06T07:45:44.565Z In(182) vmkernel: cpu64:2098147)RAL: pfioctl: error, back from pf_rm_rule
2026-01-06T07:45:44.565Z In(182) vmkernel: cpu64:2098147)addrule ioctl failed: 22
2026-01-06T07:45:44.565Z In(182) vmkernel: cpu64:2098147)Sending message to cfgAgent to raising alarm for filter import failure
2026-01-06T07:45:44.565Z In(182) vmkernel: cpu64:2098147)Failed to restore datapath state : Failure
2026-01-06T07:45:44.565Z In(182) vmkernel: cpu64:2098147)DVFilter: 1622: Couldn't find an installed filter: vNic 0, agent vmware-si
2026-01-06T07:45:44.565Z In(182) vmkernel: cpu64:2098147)DVFilter: 1727: No unrestored state left, freeing pending state for world #########
2026-01-06T07:45:44.565Z In(182) vmkernel: cpu53:#########)VMotion: 6709: 39732254719501875 D: Received all changed pages.
2026-01-06T07:45:44.565Z In(182) vmkernel: cpu64:2098147)DVFilter: 2760: World: ######### - Sent Checkpoint Restore Done for ESXIO. Status: Success
2026-01-06T07:45:44.565Z In(182) vmkernel: cpu64:2098147)DVFilter: 1756: Bringing down port due to failed DVFilter state restoration and failPolicy of FAIL_CLOSED.

 

  • On the ESXi host run nsxcli -c get firewall thresholds to see the memory usage for vsip-kentries heap:
                                Firewall Threshold Monitors
-------------------------------------------------------------------------------------------
 #          Name          Raised  Threshold  CurrValue  CurrSize   MaxSize  PeakEver  EverTime(ago)
14     vsip-kentries       True       90         91      2813 MB   3070 MB     91       17:32:34      <= problematic value

 

  • On the ESXi host, run /bin/vsipioctl getmeminfo | grep pfrkentry
  • Note that inUse is significantly high and there are failures in allocating heap space.
/bin/vsipioctl getmeminfo | grep pfrkentry
zone 8:  pfrkentry maxObj = -1, objSize = 176, alloc = 702946228, free = 686395164, inUse = 16551064, numFail = 39099, totalMem = 2912987264

 

  • Filter import appears successful in vmkernel.log
2025-06-23T15:14:31.906Z In(182) vmkernel: cpu52:2098112)Importing succeeded
2025-06-23T15:14:31.906Z In(182) vmkernel: cpu52:2098112)Filter creation report: filter = nic-#########-eth0-vmware-sfw.2, source = Import

 

  • However, GLOBAL_TABLES flag is not set on the imported filter:
/bin/vsipioctl/getkifflags -f nic-#########-eth0-vmware-sfw.2
PF_KIF_FLAG_GLOBAL_TABLES                0   <<< 0 indicates Global Tables are NOT enabled.

 

  • LOCAL flag still present in addrset:
/bin/vsipioctl/getaddrsets -f nic-#########-eth0-vmware-sfw.2 -o
addrset 004148be-3e1f-4f15-b0d8-097a1f40a0e2 {


# generation number: 0
# realization time : 2025-04-29T08:20:55
# refs: 1, 0  flags: 0x10000025 (ROOT,LOCAL,PER,ACT,ANCREF)
}

 

  • Large number of imports observed in a short span:
2025-06-23T15:13:16.067Z In(182) vmkernel: cpu52:2098112)Importing nic-#########-eth5647-vmware-sfw.1, Version 1000
...
2025-06-23T15:13:18.671Z In(182) vmkernel: cpu74:2098112)Importing nic-#########-eth0-vmware-sfw.2, Version 1100

Resolution

A permanent fix is being targeted for a future major NSX release. The fix will include major enhancements to the heap reservation and allocation algorithm, allowing it to better accommodate large bursts of filter imports without exhausting memory.

Additionally, a configuration change is planned to prevent MAC addresses from being programmed into address sets which are not used in L2 rules.

Workarounds

To remediate hosts already in an exhausted state:

  • Paced VM Evacuation (Recommended): Do not reboot the host. Instead, perform scripted, paced vMotions off the affected host (e.g., migrating 1 VM every 60 to 90 seconds). This keeps the temporary LOCAL allocation spike below the memory ceiling, allowing cfgAgent the time and headroom to process the GLOBAL_TABLES flag and self-drain the heap.

  • Concurrency Note: Parallel vMotions across the estate are safe only if each simultaneous migration targets a distinct destination host. Do not land multiple VMs on the same destination host simultaneously.

To prevent the issue from occurring:

  • Suspend Automated Bulk Operations: Avoid using automated, unthrottled vCenter Maintenance Mode evacuations and automated ESXi Live Patching campaigns, as both trigger massive concurrent filter imports.

  • Optimize Rule Application: Where possible, apply firewall rules at the DFW level instead of at the group level (Applied To = DFW) to minimize the number of unique address sets learned per host during migration.

  • Reduce Address Set Scale: Use CIDR blocks or dynamic criteria in NSGroups rather than listing individual IP addresses. Eliminate overlapping or excessively large dynamic groups to reduce the addrset footprint.

MAC Set Optimization (Only if Layer 2 rules are NOT in use): If Layer 2 rules are not used in the environment, you can modify global_macset_optimization_mode_enabled to true. This prevents MAC addresses from being programmed into address sets, significantly reducing the memory footprint.

  1. Get current settings and output to a JSON file: curl -k -u admin -H "Content-Type:application/json" -X GET https://localhost/policy/api/v1/infra/settings/firewall/security > settings.json

  2. Modify settings.json to ensure global_macset_optimization_mode_enabled is true:

    JSON
     
    {
  "idfw_enabled" : true,
  "idfw_event_log_scraper_enabled" : true,
  "idfw_loginsight_enabled" : false,
  "resource_type" : "DfwFirewallConfiguration",
  "id" : "security",
  "display_name" : "security",
  "path" : "/infra/settings/firewall/security",
  "relative_path" : "security",
  "parent_path" : "/infra",
  "remote_path" : "",
  "unique_id" : "########-####-####-####-############",
  "realization_id" : "########-####-####-####-############",
  "owner_id" : "########-####-####-####-############",
  "marked_for_delete" : false,
  "overridden" : false,
  "enable_firewall" : true,
  "disable_auto_drafts" : false,
  "global_addrset_mode_enabled" : true,
  "global_macset_optimization_mode_enabled" : true
}

  3. Run a PATCH to apply the modified setting: `curl -k -u admin -H "Content-Type:application/json" -d @settings.json -X PATCH

 

Powered by Wolken Software