DFW memory very high critical alarm is generated for host TNs, vsip-fqdn utilization on the hosts TN is above threshold.

Products

• VMware vDefend Firewall

Issue/Introduction

• Distributed Firewall (DFW) memory very high critical alarm is generated for host transport node
• vsip-fqdn utilization on the hosts TN is above threshold.
  
  Example:
  
  nsxcli_-c-getfirewallthresholds.txt

                                Firewall Threshold Monitors
-------------------------------------------------------------------------------------------
 # Name Raised Threshold CurrValue CurrSize MaxSize PeakEver EverTime(ago)
 1 cfgagent False 100 3 53 MB 1536 MB 3 1d 04:44:17
 2 dfw-cpu False 90 0 -- -- 4 31d 22:17:45
 3 dfw-session False 80 0 -- -- 55 54d 14:17:05
 4 nsx-exporter False 100 14 110 MB 768 MB 16 1d 01:29:58
 5 nsx-idps False 100 18 186 MB 1024 MB 18 1d 04:44:17
 6 vdpi False 100 41 420 MB 1024 MB 41 1d 04:44:17
 7 vsip-attr False 90 0 5 MB 1280 MB 0 --:--:--
 8 vsip-flow False 90 0 1 MB 768 MB 0 --:--:--
 9 vsip-fprules False 90 0 1 MB 2560 MB 0 --:--:--
10 vsip-fqdn True 90 99 507 MB 512 MB 99 1d 04:44:17 <---------
  
  
• You see a very high number of FDQN entries associated with NICs.
  
  Example:
  
  /bin/vsipioctl getfqdnentries -f nic-<nic>-eth0-vmware-sfw.2

Total FQDN entries: 4294965838 <---------
world 202702487 vmm0:<VM-name> vcUuid:'<uuid>'
 port 100663371 <VM-name>.eth0
  vNic slot 2
  
  
• Total Heap memory in use is very high.
  
  commands/vsipioctl_info.sh.txt

Heap: vsip-fqdn, max 512 MB
    zone 25: pffqdnippl maxObj = -1, objSize = 112, alloc = 16863613, free = 16863222, inUse = 391, numFail = 5402, totalMem = 43792
    zone 26: pffqdndomainent maxObj = -1, objSize = 360, alloc = 231765765, free = 230427419, inUse = 1338346, numFail = 85902299, totalMem = 481804560
    zone 27: pffqdnuuidpl maxObj = -1, objSize = 48, alloc = 14962167, free = 14345931, inUse = 616236, numFail = 0, totalMem = 29579328
    dynamic: objInUse = 1, memInUse = 65600, hwmMem = 65725, hwmObj = 2, alloc = 16, free = 15, avgcost = 0
    Total Heap Mem In Use = 511493280 bytes (487 MB), overhead = 8

Environment

VMware NSX 4.0.0.1

Cause

This heap usage is related to FQDN with large TTL getting programmed.

Resolution

This issue is resolved in VMware NSX 4.1.2.2
This issue is resolved in VMware NSX 4.2.0

Workaround:
Following options can be used as workaround

• Reboot affected ESXi host.
• Stop using FQDN rules for the time being.
• Putting VMs in and out of DFW exclusion list also resets vsip-fdqn utilization temporarily.