The SpectroSERVER was continually using 100% utilization of one CPU and overtime increased memory
utilization and eventually shut itself down due to an out of memory condition
Release : 21.2
Component : Spectrum Core / SpectroSERVER
Using SysInternal's Process Explorer we were able to view the call stacks on the thread that was
using 100% CPU (actually 25%, system has 4 cpus, = 100% of 1 CPU)
We keep seeing get alarm calls being processed.
Checking the localhost access logs ($SPECROOT/tomcat/logs/) on the 2 OneClick Servers
and we were seeing restful calls coming in. One OneClick was integrated with CAPM which
was making alarm calls and the other OneClick we noticed Restful subscription requests coming
in for alarms.
Checking resources on the OneClick and found that the OneClickService.exe (java on linux) was
using its max configured memory of 8192 and the other OneClick was using 4.7Gb while being
configured for the default 4098M.
The OneClick processes were out of memory causing java Garbage Collections to continually run (trying to
free up memory but the process was really out of memory). This causes slowness and the OneClicks were
not processing the requests fast enough pulling data back from the SpectroSERVER. As a result, the calls
were stacking up and putting back pressure on the SpectroSERVER
Increased the OneClick tomcat memory to 12288M (ideally 16384M or more would be better)
$SPECROOT/tomcat/bin/
Windows:
edit OneClickService.conf
Linux
edit: catalina.sh
-Xmx12288M
-Xms8192M
Next, made the same changes to the tomcat-server-config.xml
$SPECROOT/custom/common/config/tomcat-server-config.xml
-Xmx12288M
-Xms8192M
Restart OneClick tomcat.
After doing so the problem was resolved and the SpectroSERVER CPU utilization returned back to normal (3-4%)