SOI to Spectrum Alarm Not Clearing - Southbound Alert Operations doesn't work

Products

• CA Service Operations Insight (SOI)

Issue/Introduction

We are facing an issue in SOI, southbound alert operations stopped working after installing some windows patches. SOI is integrated with Service Now. When a ticket is Resolved | Closed in Service Now the associated Alert clears automatically in SOI but the same alert doesn't clear in Spectrum.

Also, when an alert is cleared manually in SOI, the same doesn't cleared in Spectrum

*** Note: Other connector alerts like UIM, SCOM also may have this behavior

Environment

SOI 4.0 & 4.2
Catalyst Container 3.4.1 & 3.4.2
Spectrum 10.2, 10.3
Spectrum Connector .259
UIM Connector 3.8, 3.9
ServiceNow London

Cause

This is due to huge records pending in ca_ssa_tags table for tagname like 'synched.pending' etc

When running below query, first 2 queries may result huge number of records, hence Southbound Synchronization takes more time than expected

• select count(*) from ca_ssa_tags where tagname='synched.pending' 
• select count(*) from ca_ssa_tags where tagname='synched.known.mdrs' 
• select count(*) from ca_ssa_tags where tagname='synchronization.inprogress'

Resolution

If above mentioned SQL queries returns huge number of records then follow these steps to address this problem

1) Stop SOI Manager, UI & All connector services 

2) Run below SQL queries against SAMStore DB

• delete from ca_ssa_tags where tagname like %synchronization.inprogress%' 
• delete from ca_ssa_tags where tagname like '%synched.pending%' 
• delete from ca_ssa_tags where tagname='synched.known.mdrs' 

3) Delete these files, logs...

• Delete all files inside "\SOI\apache-activemq\data\kahadb\"  (for SOI 4.2) Or \SOI\tomcat\webapps\activemq-web\activemq-data\ (For 4.0)
• Delete all files inside \SOI\tomcat\temp\ 
• Delete or move logs from \SOI\tomcat\logs\ folder (On SOI Manager) 
• Delete or move logs from \SOI\jsw\bin\logs\ folder (On SOI Manager) 

4) Stop Spectrum connector services

• CA SAM | SOI Integration Services
• CA SAM | SOI Event Management services

Note: For Catalyst Connectors like UIM, SCOM etc, please restart 'CA Catalyst Container' services

5) Backup & then Delete the \SOI\lib\ivy\configuration folder on the Spectrum connector machine(s) 

Note: Check if CAT is in debug on the manager machine (\SOI\tomcat\lib\log4j.xml) , if not change level to "DEBUG"
<logger name="com.ca.ssa.sor" additivity="false"> 
<level value="DEBUG" /> 
<appender-ref ref="CAT" /> 
</logger> 

6) Start SOI Manager services in the following order

• CA SOI MQ Server
• CA SOI Application service

7) Open SOI Manager debug page (http://<SOI Mananger Host>:7090/sam/debug) ->  click 'Triage Tests '
Make sure 'Triage Tests' are completed successfully, if not wait until it completes 
Start remaining SOI services on the SOI manager 

8) Start UI services on the UI server

9) Start one connector services at a time (if you start all connectors at a time, manager performance would be affected) 

• Check 'Queue Monitor' page on the manager debug page, make sure no queues are piled up 
• When all job queues are normal, then start another connector service 

10) When Spectrum connector is fully operational, test this behavior by clearing an alert in SOI Alert console
If problem still persists, collect the following logs and raise a support ticket

• A zip copy of \SOI\tomcat\logs from Manager
• A zip copy of \SOI\log folder from Spectrum connector
• A zip copy of \container\data\log & \container\ifw\log\ folders from Catalyst Connectors like UIM or SCOM
• A screenshot of the alert that was cleared in SOI with 'Source Alert ID & Alert ID with Domain ID" columns