How to change tomcat to see where REST queries are coming from that cause Spectrum SS performance problems

Products

• CA Spectrum
• DX NetOps

Issue/Introduction

One of our SpectroSERVERs is hung.  The clients in OneClick have a red border at times.  Timer latencies have been rising.  CPU is at 100%, and the SS doesn't respond to a moot trace signal.  We have REST integrations that might be causing a performance issue.  How can we log more detail and see where the REST queries are coming from?

How can the tomcat process be configured to change the local_host_access.log so that the local host access log file will show additional detail on who or where the queries are coming from?

How can the tomcat log record the IP address of the host a REST query is coming from? How can we see details about who is launching REST queries and what host they are on?

Environment

Release : All Supported Releases

Component : Spectrum Core / SpectroSERVER

Cause

REST

Resolution

To see REST data coming into Spectrum tomcat you can review the $SPECROOT/tomcat/logs/localhost_access_log*.txt files.

The localhost access logging is configured in the $SPECROOT/tomcat/conf/server.xml as this:

prefix="localhost_access_log." suffix=".txt" pattern="%l %u %t %r %s %b %D"

The patterns are explained here: 
https://tomcat.apache.org/tomcat-7.0-doc/api/org/apache/catalina/valves/AccessLogValve.html

https://tomcat.apache.org/tomcat-9.0-doc/config/valve.html#Access_Log_Valve 

Spectrum logs:  username, date, first line of request, status code of request, bytes sent, time to process the request in milliseconds

If needed, you could add other parameters from that list and cycle tomcat.  For example:

%a - Remote IP address

The localhost_access_log*.txt now shows this REST query came from xxx.xxx.xxx.xxx

xxx.xxx.xxx.xxx - user [12/May/2020:17:06:10 -0700] GET /spectrum/restful/alarms HTTP/1.1 200 35741 1344