OneClick WebApp high CPU

Products

• CA Spectrum

Issue/Introduction

today we saw very high CPU load on a OneClick server primarily used for REST subscriptions and only one OneClick WebApp session.  The load increased quite suddenly from less than 5% to over 90% CPU usage.  It was not possible to close the session regularly (closing browser window / issue "Shutdown" command from the WebApp administration console). I had to kill -9 the process, after which the CPU load quickly normalized.

Environment

Release : any

Component : Spectrum OneClick

Cause

The REST integration can have a high load on this OC server.  Java clients also show performance problems.

 

 

Resolution

This problem is only seen on the OC dedicated to REST queries and it was decided not to use it for client access.

Additional Information

During the problem out of memory errors are seen n the logs so perhaps increased the RAM dedicated to the webapp may help BUT OC JAVA was also problematic on this server, and the webapp relies on the standard tomcat so it maybe the  same root cause here and cannot be resolved via the webapp.

Right now the webapp is disabled as this server is not used for client access.

If the problem occurs on another server 

1. 
Jstack is for Jthreads dumps.
Use jstack  -F  to force a thread dump the issue is recreated.

cd <specroot>\Java\bin jstack -F <OCserver PID > > stack1.txt 
with one minute back take 3 to 4 stack outputs
 
 e.g.
cd <specroot>\Java\bin
jstack -F <OneClickservice PID > > stack1.txt
wait for a minute
jstack -F < OneClickservice PID> stack2.txt
wait for a minute
jstack -F < OneClickservice PID> stack3.txt
 
  
2. 
Jmap is for java heap memory. 
cd <specroot>\Java\bin
jmap -F <OneClickservice PID > > map1.txt
wait for a minute
jmap -F <OneClickservice PID > > map2.txt
wait for a minute
jmap -F <OneClickservice PID > > map3.txt

3. gather the tomcat log, webtomcat log and webswing log and upload for analysis.