How to increase Spectrum Tomcat Memory due to performance issues or Java Heap errors

Products

• Network Observability
• Spectrum

Issue/Introduction

In the OneClick server stdout.log or catalina.out log file java heap out of memory errors are seen. 

Here is an example of the heap errors:

java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid46200.hprof ...
Heap dump file created [4685398829 bytes in 147.345 secs]
Dec 29, 2018 10:09:34.111 - TomcatHealthWatcher stopping: full thread dump
Exception in thread "http-nio-8080-exec-80" Exception in thread "http-niohpro-8080-exec-78" Exception in thread "Timer-0" Exception in thread "http-nio-8080-ClientPoller-0" 
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "http-nio-8080-exec-76"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "http-nio-8080-exec-86"
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space

However, there are many other cases where the memory just needs to be increased.

Environment

Release: All Spectrum Releases

Resolution

Java heap out of memory message can be generated when more memory is needed than is configured. By default in Spectrum 10.2 and above, 4GB is configured.  This may not be high enough especially if you have CA PC and/or CA UIM integrated.

View the memory usage through the Administration - Web Server Memory page of the OneClick URL. 

General guidelines for increasing the tomcat memory:

If you only have 12GB of memory allocated to the OS, increase the memory to 8GB (8192)
If you have 16GB of memory allocated to the OS, increase the memory to 10GB (10240)
If you have 32GB of memory allocated to the OS, start by increasing the memory to 12GB (12288) and increase to 16GB (16384) if needed.

This also assumes you have no other applications other than Spectrum OneClick installed on the box
If tomcat continues to crash at the higher allocation and is creating an crash file (.hprof) please open a case with Support.

Editing/changing the value in the GUI updates:

On Windows the configuration file to edit is $SPECROOT\tomcat\bin\oneclickservice.conf.

On UNIX it's in $SPECROOT/tomcat/bin/Catalina.sh

Please also note that on Unix the OneClick service is called OneClick and will reflect the memory setting in the -Xmx parameter:

# ps -ef |grep -i oneclick
spectrum 266 252 0 May 25 ? 91:12 /usr/SPECTRUM/Java/bin/java -DOneClick -Xmx8192M -XX:+HeapDumpOnOutOfMemoryError

If you make the change in the GUI, the tomcat server cycles itself.

If you make the changes manually, you will have to also cycle the tomcat server for the change to take effect.

.

Additional Information

• Windows uses tomcat\bin\OneClickService.conf
• Linux uses tomcat/bin/catalina.sh
• custom/common/config/tomcat-server-config.xml is for gui 

• When you update Web Server memory via the OC UI it updates custom/common/config/tomcat-server-config.xml and that updates tomcat/bin/catalina.sh or OneClickService.conf 
• When tomcat starts it only reads tomcat/bin/catalina.sh or OneClickService.conf 
• Ultimately if you make the change under tomcat/bin/catalina.sh or OneClickService.conf and recycle tomcat it will use the new value.