DOI 1.3.2 integration with APM 10.7 - Troubleshooting, Common Issues and Best Practices

book

Article ID: 208009

calendar_today

Updated On:

Products

DX Operational Intelligence DX Application Performance Management

Issue/Introduction

The following is a list of techniques and suggestions to employ when troubleshooting APM to UIM integration issues

A) Checklist
B) What diagnostic files should I gather for Broadcom Support?

Environment

DX Operational Intelligence 1.3.x

Resolution

A) Checklist
 
Option #1) [Recommended] There is a set of OI supportability metrics available from Webview or TeamCenter > Metric View:
 
 
Here are the key metrics:
 
a) From MOM Supportability Metrics:
Expand “Custom Metric Host (Virtual) | Custom Metric Agent (Virtual) | Enterprise Manager | OI Alert Supportability Metrics | OI Alert Metrics : Ingested Alerts Per Interval
 
b) From  Collector Supportability Metrics:
Expand “Custom Metric Host (Virtual) | Custom Metric Agent (Virtual) (<collector>@<port>) | Enterprise Manager | OI Supportability Metrics | OI Metrics : Ingested Documents:Metrics Per Interval
 
Here is an example illustrating a APM to OI communication problem as you can see below, the above metric stops reporting
If needed, you define APM alerts based on those metrics:  
 
 
Option #2) Check the MOM and Collector logs and make sure there is not exception:
 
This line below indicates that connector is connected and working
 
2/05/21 11:33:54.174 PM IST [INFO] [APMInventory-0] [Manager.AppMap] Fetching OI Inventory details
 
Here are example of the error message indicating that APM to OI connectivity is not working or broken:
 
2/06/21 12:04:30.510 AM IST [ERROR] [APM.OI.MetricDataConsumer-3] [Manager.APM.OI] Error posting metrics to jarvis. 503 Service Unavailable
org.springframework.web.client.HttpServerErrorException: 503 Service Unavailable
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:121)
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:103)
at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:730)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:688)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:644)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:564)
at com.ca.apm.oi.core.MetricDataConsumer.ingest(MetricDataConsumer.java:81)
at com.ca.apm.oi.core.MetricDataConsumer.run(MetricDataConsumer.java:68)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
 
Option 3) Alternatively from ElasticSearch, you can run the below queries and see if count increases
 
http://es.<endpoint>/_cat/indices/*_apm*?v
http://es.<endpoint>/ao_apm_tt_1_1/[email protected]:desc&size=200&pretty
http://es.<endpoint>/ao_itoa_inventory_apm_1_1/[email protected]:desc&size=200&pretty
http://es.<endpoint>/ao_itoa_alarms_apm_1_1/[email protected]:desc&size=200&pretty
http://es.<endpoint>/ao_itoa_metrics_apm_2_1/[email protected]:desc&size=200&pretty
 
Here is an example:
 
 
B) What diagnostic files should I gather for Broadcom Support?
 
From APM (from both MOM and Collectors)
 
EM-HOME/logs/*
EM-HOME/config/IntroscopeEnterpriseManager.properties
 
From OI
 
Result of below queries from ElasticSearch:
http://es.<endpoint>/_cat/indices/*_apm*?v
http://es.<endpoint>/ao_apm_tt_1_1/[email protected]:desc&size=200&pretty
http://es.<endpoint>/ao_itoa_inventory_apm_1_1/[email protected]:desc&size=200&pretty
http://es.<endpoint>/ao_itoa_alarms_apm_1_1/[email protected]:desc&size=200&pretty
http://es.<endpoint>/ao_itoa_metrics_apm_2_1/[email protected]:desc&size=200&pretty
 
 

Additional Information

 

DX AIOPs - Troubleshooting, Common Issues and Best Practices
https://knowledge.broadcom.com/external/article/190815/dx-oi-troubleshooting-common-issues-and.html

Attachments