DFW Memory Usage Alarms for vsip-state Heap After Upgrade to NSX 4.2.x

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

After upgrading from a pre-4.2.x version to NSX version 4.2.x, you may begin receiving "DFW Memory usage is very high" alarms related to the vsip-state heap:

"The DFW Memory usage vsip-state on Transport node <UUID> has reached X%, which is at or above the very high threshold value of 75%."

Environment

VMware NSX 4.2.x

Cause

Starting in NSX 4.2, memory usage reporting for the vsip-state heap is based on data from the 'vsipioctl getmeminfo' command. However, this calculation incorrectly includes memory usage from zones that should be excluded:

• Zone 2: pfstatepl
• Zone 20: pfstaterttpl

These zones have independent memory allocations and are not part of the actual vsip-state heap consumption. Including them inflates the reported usage percentage.

As a result, these may be false alarms. To confirm if they are false alarms, review the below example:

Example:

You can find this output either by:

• Running 'vsipioctl getmeminfo' as root user on the ESXi host, or
• Reviewing /commands/vsipioctl_info.sh.txt within the ESXi log bundle.

Example output:

Heap: vsip-state, max 512 MB
    zone  2: pfstatepl        maxObj =    2000000, objSize =  616, alloc =       1521035595, free =       1520261425, inUse =           774170, numFail = 0, totalMem = 476888720  
    zone 10: pffrent          maxObj =       5000, objSize =   32, alloc =             7770, free =             7770, inUse =                0, numFail = 0, totalMem = 0
    zone 11: pffrag           maxObj =         -1, objSize =   88, alloc =             3525, free =             3525, inUse =                0, numFail = 0, totalMem = 0
    zone 12: pffrent6         maxObj =       5000, objSize =   40, alloc =                1, free =                1, inUse =                0, numFail = 0, totalMem = 0
    zone 13: pffrag6          maxObj =         -1, objSize =  120, alloc =                1, free =                1, inUse =                0, numFail = 0, totalMem = 0
    zone 15: pfalgport        maxObj =         -1, objSize =   80, alloc =            60470, free =            60439, inUse =               31, numFail = 0, totalMem = 2480
    zone 16: pfalgbuff        maxObj =        500, objSize = 5384, alloc =                0, free =                0, inUse =                0, numFail = 0, totalMem = 0
    zone 20: pfstaterttpl     maxObj =    2000000, objSize =   64, alloc =        233437245, free =        233278895, inUse =           158350, numFail = 0, totalMem = 10134400
    zone 21: pfsyncachepl     maxObj =         -1, objSize =  128, alloc =                0, free =                0, inUse =                0, numFail = 0, totalMem = 0
    dynamic: objInUse = 0, memInUse = 0, hwmMem = 0, hwmObj = 0, alloc = 0, free = 0, fail = 0/0, avgcost = 0
    Total Heap Mem In Use = 487025600 bytes (464 MB), overhead = 0

*Note that the totalMem value is in bytes

 

Incorrectly Calculated Usage:

Total Heap Mem In Use - 464 MB
Max Heap Size - 512 MB

464 / 512 = 0.906 (91%)

Actual Usage (excluding zones 2 and 20):

0.0024 MB (<1%)

As shown, the inflated usage is due to the inclusion of pfstatepl and pfstaterttpl, which account for nearly the entire 464 MB total.

 

Note:

The vsip-ipreputation heap also includes memory zones in its usage calculation, and is also inaccurate. All other vsip heaps in the 'vsipioctl getmeminfo' have accurate usage calculations. 

Resolution

Permanent fix:

Issue is resolved in the following versions:

• 4.2.2.2 and later
• 4.2.3.1 and later

Workaround:
These alarms are false positives and can be safely ignored. If desired, they may be suppressed or disabled.