How to change tomcat to see where REST queries are coming from that cause Spectrum SS performance problems

Products

• Spectrum
• Network Observability

Issue/Introduction

One of our SpectroSERVERs is hung.  The clients in OneClick have a red border at times.  Timer latencies have been rising.  CPU is at 100%, and the SS doesn't respond to a moot trace signal.  We have REST integrations that might be causing a performance issue.  How can we log more detail and see where the REST queries are coming from?

How can the tomcat process be configured to change the local_host_access.log so that the local host access log file will show additional detail on who or where the queries are coming from?

How can the tomcat log record the IP address of the host a REST query is coming from? How can we see details about who is launching REST queries and what host they are on?

Resolution

To see REST data coming into Spectrum tomcat you can review the $SPECROOT/tomcat/logs/localhost_access_log*.txt files.

The localhost access logging is configured in the $SPECROOT/tomcat/conf/server.xml as this:

prefix="localhost_access_log." suffix=".txt" pattern="%l %u %t %r %s %b %D"

By default we display in the $SPECROOT/tomcat/local_access_log the following information based upon the pattern variables.

%l - Remote logical username from identd (always returns '-')
%u - Remote user that was authenticated
%t - Date and time, in Common Log Format format
%r - First line of the request
%s - HTTP status code of the response
%b - Bytes sent, excluding HTTP headers, or '-' if no bytes were sent
%D - Time taken to process the request, in milliseconds

This looks like the following below:

• spectrum [01/Mar/2014:00:00:05 -0500] POST /spectrum/common/poll?requestID=27116 HTTP/1.1 200 821 0

However no local IP address or hostnames are shown.  The patterns are explained here: 
https://tomcat.apache.org/tomcat-7.0-doc/api/org/apache/catalina/valves/AccessLogValve.html

https://tomcat.apache.org/tomcat-9.0-doc/config/valve.html#Access_Log_Valve 


If needed, you could add other parameters from that list and cycle tomcat.  For example:

%a - Remote IP address

The localhost_access_log*.txt now shows this REST query came from <ip address>

<ip address> - user [12/May/2020:17:06:10 -0700] GET /spectrum/restful/alarms HTTP/1.1 200 35741 1344

Or add %A %v so the final version looks pattern="%A %v %l %u %t %r %s %b %D"
Save and after restart of the tomcat you will now see local IP address and local server name appended to all output.

<ipaddress> localhost - spectrum [14/Mar/2014:00:00:04 -0400] POST /spectrum/common/clearConfigs HTTP/1.1 200 58 0