Excessive REST queries cause Spectrum SpectroSERVER performance problems

Products

• CA Spectrum
• CA eHealth

Issue/Introduction

One of our SpectroSERVERs is hung.  The clients in OneClick have a red border at times.  Timer latencies have been rising.  CPU is at 100%, and the SS doesn't respond to a moot trace signal.  We have REST integrations that might be causing a performance issue.  How can we log more detail and see where the REST queries are coming from?

How can the tomcat process be configured to change the local_host_access.log so that the local host access log file will show additional detail on who or where the queries are coming from?

How can the tomcat log record the IP address of the host a REST query is coming from? How can we see details about who is launching REST queries and what host they are on?

Environment

Release : 10.4

Component : Spectrum Core / SpectroSERVER

Resolution

To see REST data coming into Spectrum tomcat you can review the $SPECROOT/tomcat/logs/local_host_access_log.txt files.

The localhost access logging is configured in the $SPECROOT/tomcat/conf/server.xml as this:

prefix="localhost_access_log." suffix=".txt" pattern="%l %u %t %r %s %b %D"

The patterns are explained here: 
https://tomcat.apache.org/tomcat-7.0-doc/api/org/apache/catalina/valves/AccessLogValve.html

https://tomcat.apache.org/tomcat-9.0-doc/config/valve.html#Access_Log_Valve 

Spectrum logs:  username, date, first line of request, status code of request, bytes sent, time to process the request in milliseconds

If needed, you could add other parameters from that list and cycle tomcat.  For example:

%a - Remote IP address

The local_host_acces_log.tx now shows this REST query came from 192.168.145.247

192.168.145.247 - spectrum [12/May/2020:17:06:10 -0700] GET /spectrum/restful/alarms HTTP/1.1 200 35741 1344