[SOI] - Uncontrolled restarts of SOI manager

Products

• CA Service Operations Insight (SOI)

Issue/Introduction

After implementing 3rd party .jar files to the SOI environment you may notice the following symptoms: 

- SOI Manager re-initializes itself from time to time. 
- As a consequence of this the UI servers lose the connection to the manager and therefore the SOI Console is not updated (in case it was open at the time of the re-initialization) or cannot be opened anymore. Thus a restart of the UI servers is also necessary. 
- in the log files, you can find "Deadlock messages" in the soimgr_wrapper.log : 

INFO   | jvm 1    | 2023/09/05 03:33:33 | 2023-09-05 03:33:33,774 ActiveMQ Session Task-3764 ERROR Recursive call to appended <appender_console_name>-2
INFO   | jvm 1    | 2023/09/05 03:34:54 | WrapperManager Error: Found 2 deadlocked threads!
INFO   | jvm 1    | 2023/09/05 03:34:54 | WrapperManager Error: =============================
INFO   | jvm 1    | 2023/09/05 03:34:54 | WrapperManager Error: "ActiveMQ Session Task-45642" tid=<id_number>
INFO   | jvm 1    | 2023/09/05 03:34:54 | WrapperManager Error:   java.lang.Thread.State: BLOCKED

Environment

Release : 4.2

Cause

Deadlock might be related to logging with uhd connectors or other 3rd party connectors that are using the same logging appender as the SOI.

 

Resolution

Disable 3rd party connector logging appender or remove the 3rd party .jar or application and restart services.