The following is a list of techniques and suggestions to employ when troubleshooting APM to UIM integration issues
A) Checklist
B) What diagnostic files should I gather for Broadcom Support?
DX Operational Intelligence 1.3.x
Here are the key metrics:a) From MOM Supportability Metrics:Expand “Custom Metric Host (Virtual) | Custom Metric Agent (Virtual) | Enterprise Manager | OI Alert Supportability Metrics | OI Alert Metrics : Ingested Alerts Per Intervalb) From Collector Supportability Metrics:Expand “Custom Metric Host (Virtual) | Custom Metric Agent (Virtual) (<collector>@<port>) | Enterprise Manager | OI Supportability Metrics | OI Metrics : Ingested Documents:Metrics Per IntervalHere is an example illustrating a APM to OI communication problem as you can see below, the above metric stops reportingIf needed, you define APM alerts based on those metrics:
2/05/21 11:33:54.174 PM IST [INFO] [APMInventory-0] [Manager.AppMap] Fetching OI Inventory details
2/06/21 12:04:30.510 AM IST [ERROR] [APM.OI.MetricDataConsumer-3] [Manager.APM.OI] Error posting metrics to jarvis. 503 Service Unavailableorg.springframework.web.client.HttpServerErrorException: 503 Service Unavailableat org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:121)at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:103)at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:730)at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:688)at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:644)at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:564)at com.ca.apm.oi.core.MetricDataConsumer.ingest(MetricDataConsumer.java:81)at com.ca.apm.oi.core.MetricDataConsumer.run(MetricDataConsumer.java:68)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)
http://es.<endpoint>/_cat/indices/*_apm*?vhttp://es.<endpoint>/ao_apm_tt_1_1/[email protected]:desc&size=200&prettyhttp://es.<endpoint>/ao_itoa_inventory_apm_1_1/[email protected]:desc&size=200&prettyhttp://es.<endpoint>/ao_itoa_alarms_apm_1_1/[email protected]:desc&size=200&prettyhttp://es.<endpoint>/ao_itoa_metrics_apm_2_1/[email protected]:desc&size=200&pretty
http://es.<endpoint>/_cat/indices/*_apm*?vhttp://es.<endpoint>/ao_apm_tt_1_1/[email protected]:desc&size=200&prettyhttp://es.<endpoint>/ao_itoa_inventory_apm_1_1/[email protected]:desc&size=200&prettyhttp://es.<endpoint>/ao_itoa_alarms_apm_1_1/[email protected]:desc&size=200&prettyhttp://es.<endpoint>/ao_itoa_metrics_apm_2_1/[email protected]:desc&size=200&pretty
DX AIOPs - Troubleshooting, Common Issues and Best Practices
https://knowledge.broadcom.com/external/article/190815/dx-oi-troubleshooting-common-issues-and.html