SOI UIM Connector hangs at startup and is stuck at initializing

Products

• CA Service Operations Insight (SOI)
• CA Unified Infrastructure Management On-Premise (Nimsoft / UIM)

Issue/Introduction

The UIM connector has been using high CPU and after a restart of the

Ca SOI Integration Services

The connector is stuck in ‘initializing’ for hours.

CPU is high for the java process.

Environment

Release : SOI 4.2 CU4

UIM nimsoft connector - all supported versions

Cause

Old connector log shows that the connector is fetching 504 CI's from UIM at start up:

Line 15484: 2023-01-17 06:42:21,568 INFO [CA:[email protected]_CatalystConnector] impl.NSConnector - Number of CIs returning: 504

To fetching 1.5 million in the current logs.

…

Enabled debug for connector (see bottom)

And in the transformer log we saw that the majority of CI's being sent over were application type HOSTNAME:websphere_mq

Did a SQL query on the UIM db:

SELECT COUNT(*) FROM CM_CONFIGURATION_ITEM

~1.4m

SELECT COUNT FROM CM_CONFIGURATION_ITEM where ci_name LIKE '%MERCATOR%'

~1.34m

 

So it turns out someone enabled 2 WebSphere probes that dropped 1.34 million CI's into the db and the SOI connector is trying to parse all of them and getting hung.

Resolution

Disabled the unwanted websphere probes.

Clean up the UIM database:

https://knowledge.broadcom.com/external/article/259924/

Restart the SOI connector.

Additional Information

To enable debug for the connector to see what it is getting and what it is doing with it:

 

C:\Program Files (x86)\CA\Catalyst\CatalystConnector\container\etc\log4j-ei.properties

 

Original:

# EI log

log4j.logger.com.ca.catalyst.intercepter.ei=INFO, eiinterceptor

log4j.logger.com.ca.connector.impl.util.ei=INFO, eiinterceptor

 

log4j.logger.com.ca.eventplus.catalog=WARN, Transformer

log4j.logger.com.ca.eventplus.catalog.test=WARN, Transformer

………………….

Edited for debug

# EI log

log4j.logger.com.ca.catalyst.intercepter.ei=DEBUG, eiinterceptor

log4j.logger.com.ca.connector.impl.util.ei=DEBUG, eiinterceptor

 

log4j.logger.com.ca.eventplus.catalog=DEBUG, Transformer

log4j.logger.com.ca.eventplus.catalog.test=WARN, Transformer

…………………………

 

Changes are read on-the-fly no restarts needed

 

Logs are in

C:\Program Files (x86)\CA\Catalyst\CatalystConnector\container\data\log

ei-interceptor.log

ei-transformer.log

 

C:\Program Files (x86)\CA\Catalyst\CatalystConnector\container\data\log\EIDebugData

CA_00050_hostname_Alert.log