OneClick's tomcat server runs out of memory and as a result is unable to process alarms and inventory from the spectrumgtw probe
search cancel

OneClick's tomcat server runs out of memory and as a result is unable to process alarms and inventory from the spectrumgtw probe

book

Article ID: 191430

calendar_today

Updated On:

Products

CA Spectrum

Issue/Introduction


OneClick's tomcat server runs out of memory and as a result, is unable to process alarms and inventory from the spectrumgtw probe


In the OneClick tomcat log, we can see that it quickly runs out of memory just after starting

$SPECROOT/tomcat/logs/catalina.out (stdout.log on Windows)
-----------------------------------------------------
22-May-2020 10:08:32.105 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-jsse-nio-8443"]
~
22-May-2020 10:09:41.666 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["https-jsse-nio-8443"]

~
May 22, 2020 10:09:42.263 (Thread-83) (SDIntegration) - Tomcat started. Starting NIM initilization
Rest subscription 9b079ce6-38a5-49e9-8021-a51e363136ba queue is full(30000), attempt #1
java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to java_pid7869.hprof ...
Dump file is incomplete: No space left on device

May 22, 2020 10:13:36.318 - TomcatHealthWatcher stopping: full thread dump
Exception in thread "Timer-1" java.lang.OutOfMemoryError: GC overhead limit exceeded
 at org.apache.axis2.deployment.RepositoryListener.loadOtherDirectories(RepositoryListener.java:276)
 at org.apache.axis2.deployment.RepositoryListener.checkServices(RepositoryListener.java:253)
 at org.apache.axis2.deployment.RepositoryListener.startListener(RepositoryListener.java:371)
 at org.apache.axis2.deployment.scheduler.SchedulerTask.checkRepository(SchedulerTask.java:73)
 at org.apache.axis2.deployment.scheduler.SchedulerTask.run(SchedulerTask.java:94)
 at org.apache.axis2.deployment.scheduler.Scheduler$SchedulerTimerTask.run(Scheduler.java:93)
 at java.util.TimerThread.mainLoop(Timer.java:555)
 at java.util.TimerThread.run(Timer.java:505)
-----------------------------------------------------


Environment

Release : 10.3

Component : Spectrum OneClick

Cause

In this case the OneClick tomcat had somehow reverted back to its default memory setting of 4098M. This environment is quite large
  and 4098M (4GB) is not enough memory for OneClick to function properly.


- The custom memory configuration file was not present
     $SPECROOT/custom/common/config/tomcat-server-config.xml

    Note: This file is created when you set the memory via the 'Web Server Memory' page in the OneClick Web Interface. When you click 'Save and Reconfigure'
        tomcat will create this file with the memory value set and this will take precedence over the catalina.sh setting

- The catalina.sh start script was using its default -Xmx4098M setting.

Resolution


Steps to resolve

- stopped tomcat
    cd $SPECROOT/tomcat/bin/
    ./stopTomcat.sh

- check OS memory
`top` showed that the system has 16GB of memory

- Increased the memory allocation to tomcat to 12288M (12GB)
  • cd $SPECROOT/tomcat/bin
  • edit:  catalina.sh
  • change -Xmx4098M   To   -Xmx12288M
  • save the catalina.sh
Note: The new memory value set will depend on the size of the Spectrum environment and the amount of available memory on the OneClick Server'

Note: Since the tomcat-server-config.xml was not present and we could not access the OneClick webpage we modified the catalina.sh




- Found we had a second hung tomcat java process and needed to kill that process
     ps -ef|grep -i OneClick

- started tomcat
     cd $SPECROOT/tomcat/bin
     ./startTomcat.sh


Tomcat started up and seemed to settle in around 9-10GB RES memory.

- Started the spectrumgtw and now alarms are flowing into the OneClick console from UIM as they should be