ESM service crashing frequently

Products

• VMware Smart Assurance

Issue/Introduction

Symptoms:

The Smarts ESM server is crashing frequently.

The thread from the Smarts ESM domain log available at <ESM_BASE>/smarts/local/logs which points to the cause of the crash is:  

Thread 94 (Thread 140491691652864 (LWP 13108)):

#0  0x00007fc73454149d in read () from /lib64/libpthread.so.0
#1  0x00007fc733efdcb5 in _lineread (fd=4, buf=0x7fc6c55f4780 "\020      \001", bufsz=<value optimized out>) at /work/redcurrent/DMT-9.4.1.0/13/smarts/platform/misc/posix/POSIX_stackdump.c:716
#2  0x00007fc733efde6b in sm_POSIXstacktrace (fd=-1, stdinfo=<value optimized out>, line=466, file=0x7fc734083608 "/work/redcurrent/DMT-9.4.1.0/13/smarts/platform/misc/posix/POSIX_stackdump.c") at /work/redcurrent/DMT-9.4.1.0/13/smarts/platform/misc/posix/POSIX_stackdump.c:548
#3  0x00007fc733efe8c8 in sm_LINUXstacktrace (fd=-1, stdinfo=0) at /work/redcurrent/DMT-9.4.1.0/13/smarts/platform/misc/posix/POSIX_stackdump.c:466
#4  0x00007fc733ef919a in fatalHandler (sig=11, info=0x7fc6c55f7e30, context=<value optimized out>) at /work/redcurrent/DMT-9.4.1.0/13/smarts/platform/thread/posix/sthread.c:926
#5  0x00007fc6e2228c72 in os::Linux::chained_handler () from /opt/InCharge9410/ESM/smarts/jre/lib/amd64/server/libjvm.so
#6  0x00007fc6e222eda6 in JVM_handle_linux_signal () from /opt/InCharge9410/ESM/smarts/jre/lib/amd64/server/libjvm.so
#7  0x00007fc6e22256f3 in signalHandler () from /opt/InCharge9410/ESM/smarts/jre/lib/amd64/server/libjvm.so
#8  <signal handler called>
#9  AccessorManager::setInstancePollingParametersInAccessor (instance=<value optimized out>, minimumpp=240, defaultpp=240, retries=3, timeout=0.69999999999999996, trace=0) at /work/redcurrent/DMT-9.4.1.0/13/smarts/repos/accessors/AccessorManager.c:200
#10 0x00007fc72cfef88c in MR_PollingManager::setInstancePollingParameters (this=<value optimized out>, instance=<value optimized out>, newPollParams={timeout = 0.69999999999999996, minimumpp = 240, defaultpp = 240, retries = 3, trace = -1}) at /work/redcurrent/DMT-9.4.1.0/13/smarts/repos/mr/MR_PollingManager.c:732
#11 0x00007fc72d1d1d27 in MR_PollingManagerInterface::_insert_polling_parameters (this=<value optimized out>, value=<value optimized out>) at /work/redcurrent/DMT-


ESM is crashing every time due to the JVM signal handler "JVM_handle_linux_signal" while inserting polling instrumentation objects.

The ESM log files also shows the ASL parser is looking for undefined key in the table instrumentTable[FS] and trying to instrument. 
-----------------------------------------------------------
ASL-D-ERROR_UNDEFINED_TABLE_KEY_ACCESS-Attempted to access undefined key '&2' in table '&1'
. p1: instrumentTable[FS]
-----------------------------------------------------------

Environment

Smarts - 10.1.x

Cause

This is being caused by stale entries in the RPS file.

Resolution

Restart the ESM with empty RPS and discover topology post successful restart.