Spectrum Oneclick stops processing alarms sent from the spectrumgtw (UIM Integration)
book
Article ID: 188779
calendar_today
Updated On:
Products
CA Spectrum
Issue/Introduction
Spectrum is integrated with DX Infrastructure Manager (UIM) via the spectrumgtw probe. About once a day OneClick stops processing alarms being synchronized from UIM. This results in missed alarms and tickets not being raised for the alarms.
Environment
Release : 20.2.x
Component : SPCOCK - Spectrum OneClick
Cause
In this case, both the Spectrum and DX Infrastructure Manager (UIM) installations were fairly large. The OneClick server did not have enough memory allocated and therefore ran out of memory a while after having been restarted. This caused processing issues and the inability to process alarms being generated from the spectrumgtw probe.
The OneClick tomcat log (stdout.log/catalina.out) shows out of memory errors
example: $SPECROOT/tomcat/logs/catalina.out Exception in thread "http-nio-80-Acceptor-0" java.lang.OutOfMemoryError: Java heap space at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) at org.apache.tomcat.util.net.SocketBufferHandler.<init>(SocketBufferHandler.java:41) at org.apache.tomcat.util.net.NioEndpoint.setSocketOptions(NioEndpoint.java:395) at org.apache.tomcat.util.net.NioEndpoint.setSocketOptions(NioEndpoint.java:70) at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:115) at java.lang.Thread.run(Thread.java:748) ~ ~ ~ Apr 12, 2020 00:11:41.059 - DistModelManager.addModelDomain: Error starting model watch on domain specprils1java.lang.OutOfMemoryError: Java heap space
Exception in thread "Timer-0" Apr 12, 2020 00:12:03.668 (Thread-40) (MDL) - specuser: Callback notifications were missed from landscape speclandscape5 - resynching cached alarms, attributes. java.lang.OutOfMemoryError: Java heap space 2510 java.lang.OutOfMemoryError: Java heap space ~ ~ Apr 12, 2020 00:41:10.609 - DistModelManager.addModelDomain: Error starting model watch on domain spec-mls-kdc-01java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space Apr 12, 2020 00:41:10.609 - TomcatHealthWatcher:java.lang.OutOfMemoryError: Java heap space
Resolution
Increase the memory allocated to the OneClick tomcat that the spectrumgtw is integrated with. In this specific case, the OneClick server was set to the default 4096M so we ensured that the machine had enough free memory and increased the memory to 8192M.