DFW memory very high critical alarm is generated for host TNs, vsip-fqdn utilization on the hosts TN is above threshold.
search cancel

DFW memory very high critical alarm is generated for host TNs, vsip-fqdn utilization on the hosts TN is above threshold.

book

Article ID: 322084

calendar_today

Updated On:

Products

VMware NSX Networking

Issue/Introduction

Symptoms:
- DFW is configured.
- DFW memory very high critical alarm is generated for host TNs.
- vsip-fqdn utilization on the hosts TN is above threshold.
For e.g.,
nsxcli_-c-getfirewallthresholds.txt
                                Firewall Threshold Monitors
-------------------------------------------------------------------------------------------
 # Name Raised Threshold CurrValue CurrSize MaxSize PeakEver EverTime(ago)
 1 cfgagent False 100 3 53 MB 1536 MB 3 1d 04:44:17
 2 dfw-cpu False 90 0 -- -- 4 31d 22:17:45
 3 dfw-session False 80 0 -- -- 55 54d 14:17:05
 4 nsx-exporter False 100 14 110 MB 768 MB 16 1d 01:29:58
 5 nsx-idps False 100 18 186 MB 1024 MB 18 1d 04:44:17
 6 vdpi False 100 41 420 MB 1024 MB 41 1d 04:44:17
 7 vsip-attr False 90 0 5 MB 1280 MB 0 --:--:--
 8 vsip-flow False 90 0 1 MB 768 MB 0 --:--:--
 9 vsip-fprules False 90 0 1 MB 2560 MB 0 --:--:--
10 vsip-fqdn True 90 99 507 MB 512 MB 99 1d 04:44:17 <---------

- We can see very high number of FDQN entries associated with NICs.
For e.g.,
/bin/vsipioctl getfqdnentries -f nic-<nic>-eth0-vmware-sfw.2

Total FQDN entries: 4294965838 <---------
world 202702487 vmm0:<VM-name> vcUuid:'<uuid>'
 port 100663371 <VM-name>.eth0
  vNic slot 2

- Total Heap memory in use is very high.

commands/vsipioctl_info.sh.txt

Heap: vsip-fqdn, max 512 MB
    zone 25: pffqdnippl maxObj = -1, objSize = 112, alloc = 16863613, free = 16863222, inUse = 391, numFail = 5402, totalMem = 43792
    zone 26: pffqdndomainent maxObj = -1, objSize = 360, alloc = 231765765, free = 230427419, inUse = 1338346, numFail = 85902299, totalMem = 481804560
    zone 27: pffqdnuuidpl maxObj = -1, objSize = 48, alloc = 14962167, free = 14345931, inUse = 616236, numFail = 0, totalMem = 29579328
    dynamic: objInUse = 1, memInUse = 65600, hwmMem = 65725, hwmObj = 2, alloc = 16, free = 15, avgcost = 0
    Total Heap Mem In Use = 511493280 bytes (487 MB), overhead = 8


Environment

VMware NSX 4.0.0.1

Cause

This heap usage is related to FQDN with large TTL getting programmed.

Resolution

Issue is resolved in NSX 4.1.2.2 and NSX 4.2.

Workaround:
Following options can be used as workaround,
- Reboot affected ESXi host.
- Stop using FQDN rules for the time being.
- Putting VMs in and out of DFW exclusion list also resets vsip-fdqn utilization temporarily.