OneClick performance issues such as crashing due to out of memory and high cpu usage

Products

• Network Observability
• Spectrum

Issue/Introduction

The OneClick tomcat process is consuming large amounts of memory and eventually crashes.   The cpu may also be highly utilized and consumed.

There may be other client performance issues from this as well, such as slow client response or updates delayed in the OneClick Gui.

Environment

Release : 23.x and above

Cause

Tomcat Garbage Collection has been updated in Netops 25.4.7.  Previous releases can be manually updated with the information in this document.

Resolution

The OneClick Tomcat configuration file needs to be updated to contain the following entries to enable efficient Garbage Collection.

$SPECROOT/tomcat/bin/catalina.sh on Linux
$SPECROOT/tomcat/bin/OneClickService.conf on Windows

 

On Linux, edit this java configuration line in the catalina.sh:

JAVA_OPTS="-DOneClick -server -Xmx8192M -Xms4096M -XX:+UseG1GC -XX:G1HeapRegionSize=8M -XX:G1HeapWastePercent=5 -XX:MaxGCPauseMillis=800 -XX:SurvivorRatio=8 -Dsun.net.inetaddr.ttl=60 -XX:+HeapDumpOnOutOfMemoryEr
ror -Djava.awt.headless=true -Djavax.net.ssl.trustStore=$SPECROOT/custom/keystore/cacerts -Dfile.encoding=UTF-8 -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true"

Verify how much memory is allocated to the host:

free -h

Allocate half of the total memory to tomcat - this will be the Xmx entry:

 -Xmx

For example if you have 32GB allocate 16 to tomcat:

 -Xmx16384M

Change the MaxGCPauseMillis from 800 to 250:

-XX:MaxGCPauseMillis=250 

And add these parameters after the MaxGCPauseMillis entry:

-XX:ParallelGCThreads=20
-XX:ConcGCThreads=5
-XX:InitiatingHeapOccupancyPercent=50
-Xss512k
-XX:G1ReservePercent=15

For example:

JAVA_OPTS="-DOneClick -server -Xmx16384M -Xms4096M -XX:+UseG1GC -XX:G1HeapRegionSize=8M -XX:G1HeapWastePercent=5 -XX:MaxGCPauseMillis=250 -XX:ParallelGCThreads=20 -XX:ConcGCThreads=5 -XX:InitiatingHeapOccupancyPercent=50 -Xss512k -XX:G1ReservePercent=15 -XX:SurvivorRatio=8 -Dsun.net.inetaddr.ttl=60 -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true -Djavax.net.ssl.trustStore=$SPECROOT/custom/keystore/cacerts -Dfile.encoding=UTF-8 -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true"

After making this change cycle tomcat:

Navigate to the $SPECROOT/tomcat/bin directory and run:

./stopTomcat.sh

./startTomcat.sh

 

On Windows edit the java configuration in the OneClickService.conf and change the MaxGCPauseMillis from 800 to 200:

jvm_opt=-XX:MaxGCPauseMillis=800 

And add these parameters after the MaxGCPauseMillis entry:

jvm_opt=-XX:ParallelGCThreads=20 
jvm_opt=-XX:ConcGCThreads=5 
jvm_opt=-XX:InitiatingHeapOccupancyPercent=50

For example:

jvm_opt=-XX:G1HeapWastePercent=5
jvm_opt=-XX:MaxGCPauseMillis=200
jvm_opt=-XX:ParallelGCThreads=20 
jvm_opt=-XX:ConcGCThreads=5 
jvm_opt=-XX:InitiatingHeapOccupancyPercent=50
jvm_opt=-XX:SurvivorRatio=8
jvm_opt=-Xss512k
jvm_opt=-XX:G1ReservePercent=15

 

After making these changes, cycle Tomcat

Open Windows Services.

Select the Spectrum Tomcat service and click Stop.  Once Stopped, click Start.