OneClick performance issues such as crashing due to out of memory and high cpu usage

Products

• Network Observability
• Spectrum

Issue/Introduction

The OneClick tomcat process is consuming large amounts of memory and eventually crashes.   The cpu may also be highly utilized and consumed.

There may be other client performance issues from this as well, such as slow client response or updates delayed in the OneClick Gui.

Environment

Release : 22.2 and above

Cause

Tomcat Garbage Collection is not running efficiently due to the out of box configuration.

Resolution

The OneClick Tomcat configuration file needs to be updated to contain the following entries to enable efficient Garbage Collection.

$SPECROOT/tomcat/bin/catalina.sh on Linux
$SPECROOT/tomcat/bin/OneClickService.conf on Windows

 

On Linux, edit this java configuration line in the catalina.sh:

JAVA_OPTS="-DOneClick -server -Xmx4096M -Xms4096M -XX:+UseG1GC -XX:G1HeapRegionSize=8M -XX:G1HeapWastePercent=5 -XX:MaxGCPauseMillis=800 -XX:SurvivorRatio=8 -Dsun.net.inetaddr.ttl=60 -XX:+HeapDumpOnOutOfMemoryEr
ror -Djava.awt.headless=true -Djavax.net.ssl.trustStore=$SPECROOT/custom/keystore/cacerts -Dfile.encoding=UTF-8 -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true"

Change the MaxGCPauseMillis from 800 to 200:

-XX:MaxGCPauseMillis=800 

And add these parameters after the MaxGCPauseMillis entry:

-XX:ParallelGCThreads=20 
-XX:ConcGCThreads=5 
-XX:InitiatingHeapOccupancyPercent=50

For example:

JAVA_OPTS="-DOneClick -server -Xmx16384M -Xms4096M -XX:+UseG1GC -XX:G1HeapRegionSize=8M -XX:G1HeapWastePercent=5 -XX:MaxGCPauseMillis=200 -XX:ParallelGCThreads=20 -XX:ConcGCThreads=5 -XX:InitiatingHeapOccupancyPercent=50 -XX:SurvivorRatio=8 -Dsun.net.inetaddr.ttl=60 -XX:+HeapDumpOnOutOfMemoryEr
ror -Djava.awt.headless=true -Djavax.net.ssl.trustStore=$SPECROOT/custom/keystore/cacerts -Dfile.encoding=UTF-8 -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true"

After making this change cycle tomcat:

Navigate to the $SPECROOT/tomcat/bin directory and run:

./stopTomcat.sh

./startTomcat.sh

 

On Windows edit the java configuration in the OneClickService.conf and change the MaxGCPauseMillis from 800 to 200:

jvm_opt=-XX:MaxGCPauseMillis=800 

And add these parameters after the MaxGCPauseMillis entry:

jvm_opt=-XX:ParallelGCThreads=20 
jvm_opt=-XX:ConcGCThreads=5 
jvm_opt=-XX:InitiatingHeapOccupancyPercent=50

For example:

jvm_opt=-XX:G1HeapWastePercent=5
jvm_opt=-XX:MaxGCPauseMillis=200
jvm_opt=-XX:ParallelGCThreads=20 
jvm_opt=-XX:ConcGCThreads=5 
jvm_opt=-XX:InitiatingHeapOccupancyPercent=50
jvm_opt=-XX:SurvivorRatio=8

After making these changes, cycle Tomcat

Open Windows Services.

Select the Spectrum Tomcat service and click Stop.  Once Stopped, click Start.