Spectrum Oneclick stops processing alarms sent from the spectrumgtw (UIM Integration)

Products

• CA Spectrum

Issue/Introduction

Spectrum is integrated with DX Infrastructure Manager (UIM) via the spectrumgtw probe. About once a day OneClick stops processing alarms being
  synchronized from UIM. This results in missed alarms and tickets not being raised for the alarms.

Environment

Release : 20.2.x

Component : SPCOCK - Spectrum OneClick

Cause

In this case, both the Spectrum and DX Infrastructure Manager (UIM) installations were fairly large. The OneClick server did not have enough memory allocated
  and therefore ran out of memory a while after having been restarted. This caused processing issues and the inability to process alarms being generated from
  the spectrumgtw probe.

The OneClick tomcat log (stdout.log/catalina.out) shows out of memory errors

example: $SPECROOT/tomcat/logs/catalina.out
Exception in thread "http-nio-80-Acceptor-0" java.lang.OutOfMemoryError: Java heap space
 at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
 at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
 at org.apache.tomcat.util.net.SocketBufferHandler.<init>(SocketBufferHandler.java:41)
 at org.apache.tomcat.util.net.NioEndpoint.setSocketOptions(NioEndpoint.java:395)
 at org.apache.tomcat.util.net.NioEndpoint.setSocketOptions(NioEndpoint.java:70)
 at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:115)
 at java.lang.Thread.run(Thread.java:748)
 ~
 ~
 ~
Apr 12, 2020 00:11:41.059 - DistModelManager.addModelDomain:  Error starting model watch on domain specprils1java.lang.OutOfMemoryError: Java heap space

Exception in thread "Timer-0" Apr 12, 2020 00:12:03.668 (Thread-40) (MDL) - specuser: Callback notifications were missed from landscape speclandscape5 - resynching cached alarms, attributes.
java.lang.OutOfMemoryError: Java heap space
2510 java.lang.OutOfMemoryError: Java heap space
~
~
Apr 12, 2020 00:41:10.609 - DistModelManager.addModelDomain:  Error starting model watch on domain spec-mls-kdc-01java.lang.OutOfMemoryError: Java heap space

java.lang.OutOfMemoryError: Java heap space
Apr 12, 2020 00:41:10.609 - TomcatHealthWatcher:java.lang.OutOfMemoryError: Java heap space

Resolution

Increase the memory allocated to the OneClick tomcat that the spectrumgtw is integrated with. In this specific case, the OneClick server was
  set to the default 4096M so we ensured that the machine had enough free memory and increased the memory to 8192M.