SpectroSERVER Using 100% of One CPU - getAlarmListWithAttrsNoFiltering

Products

• CA Spectrum
• DX NetOps

Issue/Introduction

The SpectroSERVER was continually using 100% utilization of one CPU and overtime increased memory 
   utilization and eventually shut itself down due to an out of memory condition

Environment

Release : 21.2

Component : Spectrum Core / SpectroSERVER

Cause

Using SysInternal's Process Explorer we were able to view the call stacks on the thread that was
   using 100% CPU (actually 25%, system has 4 cpus, = 100% of 1 CPU)

 

 

 

 

We keep seeing get alarm calls being processed. 

Checking the localhost access logs ($SPECROOT/tomcat/logs/) on the 2 OneClick Servers
   and we were seeing restful calls coming in. One OneClick was integrated with CAPM which
   was making alarm calls and the other OneClick we noticed Restful subscription requests coming
   in for alarms.

Checking resources on the OneClick and found that the OneClickService.exe (java on linux) was
    using its max configured memory of 8192 and the other OneClick was using 4.7Gb while being
    configured for the default 4098M.

The OneClick processes were out of memory causing java Garbage Collections to continually run (trying to
   free up memory but the process was really out of memory). This causes slowness and the OneClicks were
   not processing the requests fast enough pulling data back from the SpectroSERVER. As a result, the calls
   were stacking up and putting back pressure on the SpectroSERVER

Resolution

Increased the OneClick tomcat memory to 12288M (ideally 16384M or more would be better)

$SPECROOT/tomcat/bin/

Windows:
   edit OneClickService.conf
 
Linux
   edit: catalina.sh


-Xmx12288M
-Xms8192M


Next, made the same changes to the tomcat-server-config.xml

$SPECROOT/custom/common/config/tomcat-server-config.xml
-Xmx12288M
-Xms8192M


Restart OneClick tomcat.


After doing so the problem was resolved and the SpectroSERVER CPU utilization returned back to normal (3-4%)