OneClick's tomcat server runs out of memory and as a result is unable to process alarms and inventory from the spectrumgtw probe
book
Article ID: 191430
calendar_today
Updated On:
Products
CA Spectrum
Issue/Introduction
OneClick's tomcat server runs out of memory and as a result, is unable to process alarms and inventory from the spectrumgtw probe
In the OneClick tomcat log, we can see that it quickly runs out of memory just after starting
$SPECROOT/tomcat/logs/catalina.out (stdout.log on Windows) ----------------------------------------------------- 22-May-2020 10:08:32.105 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-jsse-nio-8443"] ~ 22-May-2020 10:09:41.666 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["https-jsse-nio-8443"] ~ May 22, 2020 10:09:42.263 (Thread-83) (SDIntegration) - Tomcat started. Starting NIM initilization Rest subscription 9b079ce6-38a5-49e9-8021-a51e363136ba queue is full(30000), attempt #1 java.lang.OutOfMemoryError: GC overhead limit exceeded Dumping heap to java_pid7869.hprof ... Dump file is incomplete: No space left on device
May 22, 2020 10:13:36.318 - TomcatHealthWatcher stopping: full thread dump Exception in thread "Timer-1" java.lang.OutOfMemoryError: GC overhead limit exceeded at org.apache.axis2.deployment.RepositoryListener.loadOtherDirectories(RepositoryListener.java:276) at org.apache.axis2.deployment.RepositoryListener.checkServices(RepositoryListener.java:253) at org.apache.axis2.deployment.RepositoryListener.startListener(RepositoryListener.java:371) at org.apache.axis2.deployment.scheduler.SchedulerTask.checkRepository(SchedulerTask.java:73) at org.apache.axis2.deployment.scheduler.SchedulerTask.run(SchedulerTask.java:94) at org.apache.axis2.deployment.scheduler.Scheduler$SchedulerTimerTask.run(Scheduler.java:93) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) -----------------------------------------------------
Environment
Release : 10.3
Component : Spectrum OneClick
Cause
In this case the OneClick tomcat had somehow reverted back to its default memory setting of 4098M. This environment is quite large and 4098M (4GB) is not enough memory for OneClick to function properly.
- The custom memory configuration file was not present $SPECROOT/custom/common/config/tomcat-server-config.xml
Note: This file is created when you set the memory via the 'Web Server Memory' page in the OneClick Web Interface. When you click 'Save and Reconfigure' tomcat will create this file with the memory value set and this will take precedence over the catalina.sh setting
- The catalina.sh start script was using its default -Xmx4098M setting.
Resolution
Steps to resolve
- stopped tomcat cd $SPECROOT/tomcat/bin/ ./stopTomcat.sh
- check OS memory `top` showed that the system has 16GB of memory
- Increased the memory allocation to tomcat to 12288M (12GB)
cd $SPECROOT/tomcat/bin
edit: catalina.sh
change -Xmx4098M To -Xmx12288M
save the catalina.sh
Note: The new memory value set will depend on the size of the Spectrum environment and the amount of available memory on the OneClick Server'
Note: Since the tomcat-server-config.xml was not present and we could not access the OneClick webpage we modified the catalina.sh
- Found we had a second hung tomcat java process and needed to kill that process ps -ef|grep -i OneClick
- started tomcat cd $SPECROOT/tomcat/bin ./startTomcat.sh
Tomcat started up and seemed to settle in around 9-10GB RES memory.
- Started the spectrumgtw and now alarms are flowing into the OneClick console from UIM as they should be