Spectrum OneClick Java uses 300% CPU causing Tomcat crash

Products

• CA Spectrum
• CA eHealth

Issue/Introduction

Java utilizes 300%+ of the CPU, the server then becomes unresponsive.
Memory usage on the server is not high.
This is a OneClick only server with low memory usage of the 16 GB installed. 
It is integrated with CAPC and receives REST calls to pull alarms for a 3rd party product.

Environment

Release : 10.3

Component : Spectrum Core / SpectroSERVER

Cause

While the server has 16 GB of memory, only the default of 4 GB was dedicated to Tomcat.
This needs to be increased to a minimum of 8GB for most CAPC integrations.
As CAPC integrations have become so demanding, we are now recommending 32 GB for integrated OneClick servers with 16 GB dedicated to tomcat.

Resolution

In this situation, the custom REST calls had a huge cost on memory and were generating hprofs of 10 GB in size, so 12 GB  for tomcat was recommended.
The Tomcat memory is configured from the OneClick web page "Web Server Memory".
This can be verified from the following files

windows:
\Spectrum\custom\common\config\tomcat-server-config.xml

Linux:
\Spectrum\custom\common\config\tomcat-server-config.xml
and
/Spectrum/tomcat/bin/catalina.sh