Tomcat crashing and running out of Java Heap memory
To provide a work around if Smarts Tomcat is having issue with Java Heap memory.
Environment
SMARTS-10.1.x
Cause
Causes can be due to Watch4net collector getting turned off, or more data flowing and the publishing queue and SAM notifications exceeds java heap memory.
Resolution
Increase java heap size for EDAA tomcat in Smarts.
First you need to run a check for Total Available Free Memory on the Linux VM using free -m -t that is hosting Smarts SAM and EDAA.
P.S: Do not proceed if the output show Swap Used greater then zero.
Also the Total Free memory is shown highlighted here as 48 GB, make sure this number twice the proposed Java heap increase, otherwise if your server starts using Swap memory, Smarts may become starved for memory resources and disk resources and crash.
The java heap used by sm_tomcat can be adjusted by making modification to the sm_tomcat.options file located under <installation path>/SAM/smarts/bin/system/ directory.
You can increase the memory by making change to the -Xmx value.
Once done, save the file and restart the sm_tomcat service/.
Update to: SM_CATALINA_OPTS=${SM_CATALINA_OPTS:--Xms1g -Xmx6g -XX:PermSize=512m -XX:+HeapDumpOnOutOfMemoryError -XX:+UseLargePages}
If you still have issues after this update, then you will need to determine what changed in the Smarts SAM environment or the Watch4net collector that is causing the memory growth and the EDAA tomcat to run out of Java heap memory and submit a SR to VMware by Broadcom.