Smarts SAM: Trap Exploder and Memory Leak

Products

• VMware Smart Assurance

Issue/Introduction

Symptoms:


Memory utilization causes alerts and potentially crash the adapter.

Trap Exploder experiences Trap spike and memory grows to well over 100%. Trap Exploder goes back to normal but memory does not.

Environment

VMware Smart Assurance - SMARTS

Cause

This might be the behavior of memory allocator (in this case ptmalloc). The memory allocator can return free memory to system only in big chucks. However, when a chunk has holes in it e.g., the chunk is only freed partially, it will stay in the allocator bins, and still counted as used by the application   this is memory fragmentation issue.

Resolution

It is recommended to increase the Thread count for Trap Exploder from 1 to 8 and restart the adapter.

You can see your thread usage by executing the following command:

dmctl -s <Trap-Exploder> exec dmdbug --threads --output=RA_trap-threads

Also, to see how many threads are running open:
TA (TrapAdapter) & TE(TrapEploder)
<BASEDIR>Smarts/conf/trapd/trapd.conf

Example:
<BASEDIR>Smarts/bin sm_edit ../conf/trapd/trapd.conf

An excerpt from trapd.conf (top of the page)
#
# Lines starting with "#" are comments.
#
# Following is a list of parameters with a brief description.
# More information is available at the end of this file.
.
.

#
# Set the parameters here.
PORT: 9000
#WINDOW: 10
#THREADS: 1
#ASCII: FALSE

Additional Information

If increasing the thread count from 1 to 8, didn't resolve the issue, generate an sm_monitor, and open a Service Request.:

Usage:
sm_monitor -s <server> [options... ]

Options:
                -b - Define new broker
                -m -  mem|correlation|run-all
                        mem - memory growth or crash data collection ( default )
                        correlation - Network device or interface correlated incorrect status
                        run-all - run both mem and correlation data collection
                -S - Process Type ( adapter/trapd, default is server )
                -t - Turn on truss monitoring
                -z - Turn on stacktrace ouput
                -k - Turn on lock table output
                -w - Define sleeptime between loops (Default 25 for Memory only)
                -l - Define number of loops (Default 1)
                -h - Displays this usage screen
                -v - Version

Example:
<BASEDIR>/Smarts/bin sm_monitor -s <INCHARGE> -mSzk