DX OI integration with UIM - Troubleshooting guidelines

Products

• DX Operational Intelligence

Issue/Introduction

The following is a list of techniques and suggestions to employ when troubleshooting DOI-UI oi_connector integration common installation and configuration issues.

1. A) Checklist
2. B) What diagnostic files should I gather for Broadcom Support?

Environment

DOI 1.3.x

UIM OI_connector

Resolution

A) Checklist

Assumption:  Elastic router = es.muntest000471

From UIM side:

1. Check that the expected alerts in OI are visible in Infrastructure Management, this to confirm if the issue is related to UIM, Elastic or OI.

From OI_connector

1. Ensure configuration is correct

Go to the UIM Admin console > OI_connector configuration page, use the verify (Elastic/Jarvis/TenatID) options to verify connectivity

For example:

 

2. Review the oi_connector log, default location : C:\Program Files (x86)\Nimsoft\probes\gateway\oi_connector\oi_connector.log

Check for possible error messages, here is an example of a message indicating a connectivity issue:

Jan 27 17:19:46:877 [QOS_PROCESSOR_THREAD, oi_connector] Exception while executing doPost request to Jarvis url: http://jarvis.gvecsl6636ipm/ingestion :: org.apache.http.NoHttpResponseException: jarvis.gvecsl6636ipm:80 failed to respond

Check that data is being ingested correctly in elastic search for the below strings

“Total no of” QoS posted on jarvis via post”

“Total no of UIM Alarms posted on jarvis via post”

“Total no of UIM Groups posted on jarvis via post”

 

Here are some examples:

Dec 09 09:23:12:176 [UIM_OPEN_ALARM_PROCESSOR_THREAD, oi_connector] Total no of UIM Alarms posted on jarvis via post : 2

Dec 09 09:23:12:177 [DEVICE_INVENTORY_PROCESSOR_THREAD, oi_connector] Total no of UIM Device Inventory posted on jarvis via post : 2

Jan 28 08:21:31:017 [QOS_PROCESSOR_THREAD, oi_connector] Total no of QoS posted on jarvis via post : 6

 

If existing messages in INFO logs doesn’t help you can try to switch to TRACE logging for a short period of time (5 to 10 minutes) to find out about details of the data being ingested into Jarvis.

You can enable this from the UIM Admin console | <select the server where OI connector is deployed> | configure

Here is an example:

3. You can also try to restart the OI_connector from IM

From ElasticSearch

Verify if data (metrics, alerts) is being ingested in ES

1. a) http://es.<servername>/_cat/indices/*uim*?v

to have a global view of the indices activity, for example:

When troubleshooting you can monitor activity in ao_itoa_alarms_uim_1_1 (doc.count) for example to confirm whether alarms are being stored in ES as expected.

1. b) http://es.<severname>/ao_itoa_alarms_all_1_1/_search?pretty&sort=timestamp:desc&size=500

to see alerts being ingested, for example:

From OI UI

You can monitor activity in Elastic by looking at the indices from OI / Logs option, select any of the available UIM indices.

Make sure to select the right date range, here is an example:

 

B) What diagnostic files should I gather for Broadcom Support?

From UIM:

Screenshots illustrating the discrepancy

For example, if alarms are missing, screenshot from IM illustragint the alarms that you are expecting to see in OI UI

 

From OI_Connector:

- Switch from INFO to TRACE logging level for a few minutes and gather the logs as indicated above

Collect:

- C:\Program Files (x86)\Nimsoft\probes\gateway\oi_connector/logs/*

- C:\Program Files (x86)\Nimsoft\probes\gateway\oi_connector/ oi_connector.cfg 

 

From Elastic

-Result of below queries:

http://es.<servername>/_cat/indices/*uim*?v

http://es.<severname>/ao_itoa_alarms_all_1_1/_search?pretty&sort=timestamp:desc&size=500

 

Additional Information

https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/it-operations-management/digital-operational-intelligence/1-3-2/installing/integration/integrate-ca-products/add-ca-uim.html