DOI 1.3.2 integration with APM 10.7 - Troubleshooting, Common Issues and Best Practices

Products

• DX Operational Intelligence
• DX Application Performance Management

Issue/Introduction

The following is a list of techniques and suggestions to employ when troubleshooting APM to UIM integration issues

A) Checklist
B) What diagnostic files should I gather for Broadcom Support?

Environment

DX Operational Intelligence 1.3.x

Resolution

A) Checklist

 

Option #1) [Recommended] There is a set of OI supportability metrics available from Webview or TeamCenter > Metric View:

 

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/application-performance-management/10-7/integrating/Integrating-CA-APM-OnPrem-with-DX-Operational-Intelligence-SaaS---Hotfix-43.html#concept.dita_e60b6158-0eea-4cc8-9f2b-8d77de552305_Metrics

 

Here are the key metrics:

 

a) From MOM Supportability Metrics:

Expand “Custom Metric Host (Virtual) | Custom Metric Agent (Virtual) | Enterprise Manager | OI Alert Supportability Metrics | OI Alert Metrics : Ingested Alerts Per Interval

 

b) From  Collector Supportability Metrics:

Expand “Custom Metric Host (Virtual) | Custom Metric Agent (Virtual) (<collector>@<port>) | Enterprise Manager | OI Supportability Metrics | OI Metrics : Ingested Documents:Metrics Per Interval

 

Here is an example illustrating a APM to OI communication problem as you can see below, the above metric stops reporting

If needed, you define APM alerts based on those metrics:  

 

 

Option #2) Check the MOM and Collector logs and make sure there is not exception:

 

This line below indicates that connector is connected and working

 

2/05/21 11:33:54.174 PM IST [INFO] [APMInventory-0] [Manager.AppMap] Fetching OI Inventory details

 

Here are example of the error message indicating that APM to OI connectivity is not working or broken:

 

2/06/21 12:04:30.510 AM IST [ERROR] [APM.OI.MetricDataConsumer-3] [Manager.APM.OI] Error posting metrics to jarvis. 503 Service Unavailable

org.springframework.web.client.HttpServerErrorException: 503 Service Unavailable

at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:121)

at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:103)

at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)

at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:730)

at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:688)

at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:644)

at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:564)

at com.ca.apm.oi.core.MetricDataConsumer.ingest(MetricDataConsumer.java:81)

at com.ca.apm.oi.core.MetricDataConsumer.run(MetricDataConsumer.java:68)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

 

Option 3) Alternatively from ElasticSearch, you can run the below queries and see if count increases

 

http://es.<endpoint>/_cat/indices/*_apm*?v

http://es.<endpoint>/ao_apm_tt_1_1/[email protected]:desc&size=200&pretty

http://es.<endpoint>/ao_itoa_inventory_apm_1_1/[email protected]:desc&size=200&pretty

http://es.<endpoint>/ao_itoa_alarms_apm_1_1/[email protected]:desc&size=200&pretty

http://es.<endpoint>/ao_itoa_metrics_apm_2_1/[email protected]:desc&size=200&pretty

 

Here is an example:

 

 

B) What diagnostic files should I gather for Broadcom Support?

 

From APM (from both MOM and Collectors)

 

EM-HOME/logs/*

EM-HOME/config/IntroscopeEnterpriseManager.properties

 

From OI

 

Result of below queries from ElasticSearch:

http://es.<endpoint>/_cat/indices/*_apm*?v

http://es.<endpoint>/ao_apm_tt_1_1/[email protected]:desc&size=200&pretty

http://es.<endpoint>/ao_itoa_inventory_apm_1_1/[email protected]:desc&size=200&pretty

http://es.<endpoint>/ao_itoa_alarms_apm_1_1/[email protected]:desc&size=200&pretty

http://es.<endpoint>/ao_itoa_metrics_apm_2_1/[email protected]:desc&size=200&pretty

 

 

Additional Information

 

DX AIOPs - Troubleshooting, Common Issues and Best Practices
https://knowledge.broadcom.com/external/article/190815/dx-oi-troubleshooting-common-issues-and.html