We are not getting alarm from CAPC console to Spectrum OneClick console

Products

• DX NetOps
• CA Spectrum

Issue/Introduction

When the thresholds are breached of performance in CAPC console it's trigger the events on CAPC console. But that events are not showing in the OneClick console.

The following errors were found in the Tomcat log file ($SPECROOT/tomcat/logs/catalina.out file):

aug 24, 2022 10:39:14 AM (PoolThread-239: GlobalPool => RubiconEventPoller) (org.apache.axis2.transport.http.HTTPSender) - (INFO) - Unable to send to url[http://PC_hostname_FQDN:8281/EventManager/EventManagerWS.asmx]
java.net.SocketTimeoutException: Read timed out

Aug 24, 2022 10:39:14.429 (PoolThread-239: GlobalPool => RubiconEventPoller) (NetqosEventLog) - (ERROR) - RubiconEventPoller (SS_hostname(0x2000000)) Failed to retrieve events:
Caused by: org.apache.axis2.AxisFault: Read timed out

Caused by: java.net.SocketTimeoutException: Read timed out

Environment

Release : Any
Component : Spectrum Integrations

Cause

It seems the problem is it found 34328 events. OneClick doesn't send a limit. So we (PC/PM) try and build a SOAP body of 34328 events. It must fail to fully build that, but tries and sends some of it. Not sure how much.

In the /opt/CA/PerformanceCenter/EM/logs/wrapper-20220824.log file:

INFO   | jvm 1    | 2022/08/24 09:17:21 |       | getEventListOrUpdatesWithAllProperties, ItemID=116, utcStartTime=1661066048, utcEndTime=1661152448: 34328 events

According to the log message, it's trying to get a big block of events, because it hasn't completed the call since the start timestamp.

And the subsequent outgoing body has "Response-Code 500" meaning server error. But seems to have results...

INFO   | jvm 1    | 2022/08/24 09:17:21 | INFO  | qtp2084951320-21         | 2022-08-24 09:17:21,712 | org.apache.cxf.binding.soap.interceptor.Soap12FaultOutInterceptor
INFO   | jvm 1    | 2022/08/24 09:17:21 |       | class org.apache.cxf.binding.soap.interceptor.Soap12FaultOutInterceptor$Soap12FaultOutInterceptorInternalapplication/soap+xml
INFO   | jvm 1    | 2022/08/24 09:17:21 | INFO  | qtp2084951320-21         | 2022-08-24 09:17:21,712 | org.apache.cxf.services.EventManagerWS.EventManagerWSSoap.EventManagerWSSoap
INFO   | jvm 1    | 2022/08/24 09:17:21 |       | Outbound Message
INFO   | jvm 1    | 2022/08/24 09:17:21 | ---------------------------
INFO   | jvm 1    | 2022/08/24 09:17:21 | ID: 40836
INFO   | jvm 1    | 2022/08/24 09:17:21 | Response-Code: 500
INFO   | jvm 1    | 2022/08/24 09:17:21 | Encoding: UTF-8
INFO   | jvm 1    | 2022/08/24 09:17:21 | Content-Type: application/soap+xml
INFO   | jvm 1    | 2022/08/24 09:17:21 | Headers: {}
INFO   | jvm 1    | 2022/08/24 09:17:21 | Messages: (message truncated to 49152 bytes)
INFO   | jvm 1    | 2022/08/24 09:17:21 | 
INFO   | jvm 1    | 2022/08/24 09:17:21 | Payload: <soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope"><soap:Body><ns5:GetEventListOrUpdatesWithAllPropertiesResponse xmlns:ns2="http://netqos.com/nqevents" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:ns4="http://netqos.com/nqevents/filter" xmlns:ns5="http://netqos.com/nqevents/EventManager" xmlns:ns6="http://schemas.xmlsoap.org/soap/encoding/"><ns5:GetEventListOrUpdatesWithAllPropertiesResult><ns5:EventDataPlusAllProperties><ns2:EventID>2091628</ns2:EventID><ns2:Source>Data [email protected]</ns2:Source><ns2:Type>ThresholdViolation</ns2:Type><ns2:SubType>Raised</ns2:SubType><ns2:Category>4</ns2:Category><ns2:OpenTimeStamp>1661065500</ns2:OpenTimeStamp><ns2:CurrentTimeStamp>1661066109</ns2:CurrentTimeStamp><ns2:CurrentState>0</ns2:CurrentState><ns2:Severity>0</ns2:Severity><ns2:Description>A Threshold Violation event has been raised. (Profile Name: Device Polling Statistics, Rule Name: Data Collector Dropped Poll 

Resolution

NOTE: Starting from DX NetOps Spectrum 21.2.4, the default root password for MySql is "MySqlR00t". For DX NetOps Spectrum versions prior to 21.2.4, the default root password is "root". In the following MySql commands, replace <PASSWD> with the root password for your DX NetOps Spectrum version.

- Log into the OneClick system as the user that owns the Spectrum installation

- If on Windows, start a bash shell by running "bash -login"

- Stop the Spectrum Tomcat service

- cd to the $SPECROOT/mysql/bin directory and enter the following command to log into mysql:

./mysql --defaults-file=../my-spectrum.cnf -uroot -p<PASSWD> netqos_integ -A

- Enter the following command to truncate the netqos_integ.pollers table:

truncate table pollers;

- Start the Spectrum Tomcat service