DX OI integration with NetOps PM and NFA - OI Connector 1.5 Troubleshooting
Article ID: 226883
Updated On:
Products
Issue/Introduction
The following is a list of techniques and suggestions to employ when troubleshooting OI connector issues
Environment
DX NetOps OI Connector 1.5 ONLY
NOTE: OIConnector 1.5 is supported with DX NetOps Performance Management 20.2.3 or lower
Resolution
Table of Contents
A) ARCHITECTURE
B) CHECKLIST
CHECK#1 : Verify the OI connector configuration
a) Check Compatibility
b) Check OI values entered during installation are correct
| APM Gateway Hostname |
If OI 20.2.x: - If OI SaaS or 21.x: |
| APM Gateway Security Token |
If OI 20.2.x: Option 1) Login to the Tenant using and Admin account, go to DX Application Performance Management (APM)> Settings> Security > Generate New Token > select “Tenant” or "Agent" Token If OI SaaS or 21.x |
| Jarvis Server Hostname |
If OI 20.2.x): - If Openshift : oc get routes -ndxi| grep nginx If OI SaaS or 21.x: |
| Tenant ID |
If OI 20.2.x: a) Obtain the "ElasticSearch endpoint" - If Openshift: oc get routes -n<dxi-namespace> | grep jarvis b) Open your browser, list all available tenants: If OI SaaS or 21.x |
CHECK#2 : Check services
a) Check that the OI Connector services are up and running
service caperfcenter_oiconnector status
service caperfcenter_oiagent status
b) Check that the OI Connector services in NetOps console
If OI SaaS or 21.x:
Go to DX NetOps web page > Administration > Data Sources / System Status page
Locate the "OI Connector" section
Verify Status = Normal
If OI 20.2.x:
the OI Connector will report the status as failed. You can ignore the status.
CHECK#3 : Check logs
a) Review OIConnector logs : <OIConnector-HOME>/logs
- OIConnector.log : main log file
- OIAgent*.log : NFA, ADA data collection activity
b) Enable DEBUG logging:
- OIConnector logging: <OIConnector-HOME>/conf/log4j.xml
Open ./conf/log4j.xml, change logging level from INFO to DEBUG as below:
...
<!-- ***** Root Logger definition ***** -->
<root>
<level value="DEBUG"/>
<appender-ref ref="console"/>
<appender-ref ref="complete" />
</root>
- OI Agent service logging: <OIConnector-HOME>/conf/agent-wrapper.conf
Uncomment the below line:
#wrapper.app.parameter.2=-Ssupport
You need to restart the oi OI Agent service:
service caperfcenter_oiagent restart
c) Example of common errors or exceptions :
USE-CASE#1 :Problem with Data Aggregator impacting metrics gathering
ERROR [2020-09-12 22:58:41,943] [pool-2-thread-6] [OpenAPIInventoryQueryInfHelper] - [EVENT UNSPECIFIED Anonymous:[email protected] -> /com.ca.im.oinet.connector.sources.da.inventory.OpenAPIInventoryQueryInfHelper] OpenAPI Inventory Query Failed with Status: 500
..
INFO [2020-09-11 20:34:33,773] [pool-7-thread-5] [StatusHolder] - [EVENT UNSPECIFIED Anonymous:[email protected] -> /com.ca.im.oinet.connector.status.StatusHolder] Updating OI_CONNECTOR_NAME status information, STATUS [UP -> UP] HEALTH [NORMAL -> FAILED] Unable to contact OpenAPI for data
Or DX NetOps web page > Administration > Data Sources / System Status page: "Unable to connect OpenAPI for data"
Recommendation:
Restart Data Aggregator (DA), see https://knowledge.broadcom.com/external/article?articleId=190659
USE-CASE #2 : Problem with Jarvis(nginx) and/or apmservices-gateway endpoints
INFO [2020-09-11 20:00:32,988] [pool-2-thread-10] [StatusHolder] - [EVENT UNSPECIFIED Anonymous:[email protected] -> /com.ca.im.oinet.connector.status.StatusHolder] Updating OI_CONNECTOR_NAME status information, STATUS [UP -> UP] HEALTH [NORMAL -> FAILED] Unable to contact OI Platform to send data to
ERROR [2020-09-11 20:00:32,988] [pool-2-thread-10] [RemoteDataConnectionImpl] - [EVENT UNSPECIFIED Anonymous:[email protected] -> /com.ca.im.oinet.connector.sources.RemoteDataConnectionImpl]Error posting documents to Jarvis index(itoa_groups_capm): 503_
ERROR [2020-09-11 20:00:32,989] [pool-2-thread-10] [GroupTaskImpl] - [EVENT UNSPECIFIED Anonymous:[email protected] -> /com.ca.im.oinet.connector.task.group.GroupTaskImpl] Unable to push 27 groups to data sink.
ERROR [2020-09-11 20:00:33,607] [pool-2-thread-3] [TASGroupTask] - [EVENT UNSPECIFIED Anonymous:[email protected] -> /com.ca.im.oinet.connector.task.group.TASGroupTask] Failed ingesting groups to TAS for CAPC tenant id : Coke_test Error: 503
Recommendation:
Verify that the nginx and apm-gateway endpoints are correct and are available
USE-CASE #3 : CAPM user password expired, changed or not valid.
ERROR [2020-11-27 12:42:12,213] [WrapperSimpleAppMain] [OIIntegration] - [EVENT UNSPECIFIED Anonymous:[email protected] -> /com.ca.im.oinet.connector.OIIntegration] No response from webservice - unable to configure data sources
WARN [2020-11-27 12:42:12,230] [WrapperSimpleAppMain] [OIIntegration] - [EVENT UNSPECIFIED Anonymous:[email protected] -> /com.ca.im.oinet.connector.OIIntegration] Unable to determine CA Performance Center version
Recommendation:
Update the <OIConnector-HOME>/conf/config.xml with the new encoded password, see: https://knowledge.broadcom.com/external/article/204144/dx-oi-oiconnector-not-connecting-when-c.html
d) Search for common keywords: "CLIENT_SUMMARY_NASS" , “Successfully", "Started", "JARVIS_INGEST_RECORD_COUNT"
Below are some examples:
...
INFO [2020-09-13 09:24:50,587] [pool-2-thread-4] [InventoryTaskImpl] - [EVENT UNSPECIFIED Anonymous:[email protected] -> /com.ca.im.oinet.connector.task.inventory.InventoryTaskImpl] Successfully ingested inventory to TAS for CAPC tenant id : Coke_test
..
INFO [2020-09-13 09:18:14,105] [pool-2-thread-2] [RemoteDataConnectionImpl] - [EVENT UNSPECIFIED Anonymous:[email protected] -> /com.ca.im.oinet.connector.sources.RemoteDataConnectionImpl] JARVIS_INGEST_RECORD_COUNT : 27
INFO [2020-09-13 09:18:15,190] [pool-2-thread-7] [TASGroupTask] - [EVENT UNSPECIFIED Anonymous:[email protected] -> /com.ca.im.oinet.connector.task.group.TASGroupTask] Successfully ingested groups to TAS for CAPC tenant id : Coke_test
..
INFO [2020-09-13 09:11:31,481] [pool-3-thread-1] [PersistentRegistrationCache] - Successfully loaded760 metric registrations from /opt/CA/OIConnector/conf/MetricRegistrationCache-F1B889C8-4BB8-4860-BB22-447D0EEA56B0.ser
INFO [2020-09-13 09:11:31,492] [pool-3-thread-1] [NASSClient] - Started NASS Client.
..
INFO [2020-09-13 09:20:35,748] [WrapperSimpleAppMain] [ServerConnector] - Started ServerConnector@76f3fc3b{HTTP/1.1,[http/1.1]}{0.0.0.0:8782}
CHECK#4 : Check Alarms, Metrics and Topology data from DX OI UI
a) Metrics (NASS)
Go to Performance:
In SaaS, 21.x
In 20.2.x on premise:
b) Inventory and Topology (TAS)
Go to DX OI > Services > Create a new Service
From Add Elements,, select Network > Device Names, you should be able to see your NetOps devices, below an example:
CHECK#5 : Check the Alarms, Metrics, Topology data using Elastic and TAS/NAS REST APIs
** This section is valid for DX On Premise 20.x, 21.x versions only, if you are using DX OI SaaS, contact Broadcom Support for assistance **
a) Alarms(ElasticSearch)
For details how to query elasticsearch refer to : https://knowledge.broadcom.com/external/article/207215
1) List all the UIM product indices:
http://es.<servername>/_cat/indices/*capm*?v
For example:
http://es.munqa001493.bpc.broadcom.net/_cat/indices/*capm*?v
Check that doc.count and size columns values increases over the time.
2) Check the content of a specific index:
http://es.<severname>/<index-name>/_search?pretty&[email protected]:desc&size=500
For example:
http://es.munqa001493.bpc.broadcom.net/ao_itoa_groups_capm_1_1/_search?pretty&[email protected]:desc&size=500
You can use https://www.epochconverter.com/ to convert values from @timestamp field to human-readable format,
b) Inventory and Topology (TAS)
Option 1: Use DX Dashboard > AIOps Inventory source, see: https://knowledge.broadcom.com/external/article/226599
Option 2: User REST APIs:
Open Postman (you can download postman from https://www.postman.com/downloads/)
POST API End Point to check TAS data for UIM inventory:
http://<APMServices Gateway Host>/tas/graph/query
For example:
http://apmservices-gateway.munqa001493.bpc.broadcom.net/tas/graph/query
Headers:
Content-Type: application/json
Authorization: Bearer <Tenant Token>
Body:
{
"filter": {
"op": "JOIN",
"input": {
"op": "AND",
"input": [
{
"op": "ATTRIBUTE",
"expressions": [
{
"name": "Product",
"values": [
"CAPC"
]
}
]
}
]
}
},
"universe": null,
"version": null,
"time": 0,
"stitchingEnabled": true,
"includeStatus": true
}
Expected Result: you should see all new vertices added to TAS
c) Metrics(NASS)
Option 1: Use DX Dashboard > AIOps Metadata source
Option 2: User REST APIs:
Open Postman (you can download postman from https://www.postman.com/downloads/)
POST API End Point to check NASS Metric Metadata matching a pattern
http://<APM Service Gateway Host>/metadata/queryMetric
For example:
http://apmservices-gateway.munqa001493.bpc.broadcom.net/metadata/queryMetric
Headers:
Content-Type: application/json
Authorization: Bearer <Tenant Token>
Body:
{
"size": 10000,
"specifier": {
"op": "SPEC",
"sourceNameSpecifier": {
"op": "REGEX",
"pattern": "(.*)NetOps\\|CAPM(.*)|(.*)NetOps\\|ADA(.*)|(.*)NetOps\\|NFA(.*)"
},
"attributeNameSpecifier": {
"op": "ALL"
}
}
}
Expected Result: you should see all new vertices added to NAS
CHECK#6 : Verify Jarvis, Elastic, Zookeeper and Kafka
** This section is valid for DX On Premise 20.x, 21.x versions only, if you are using DX OI SaaS, contact Broadcom Support for assistance **
AIOps - Jarvis (kafka, zookeeper, elasticSearch) Troubleshooting
C) WHAT FILES SHOULD I COLLECT FOR BROADCOM SUPPORT?
If you still need assistance, contact Broadcom Support (https://support.broadcom.com/) and provide the below information:
a) DEBUG oi_connector logs
<OIConnector>/logs/*
<OIConnector>/conf/config.xml
b) services status:
service caperfcenter_oiconnector status
service caperfcenter_oiagent status
If you are using DX OI On Premise (20.x, 21.x):
a) cluster and pods status:
kubectl get pods -n<namespace>
kubectl describe nodes -n<namespace>
kubectl get events -n<namespace>
b) from ElasticSearch
- collect result of below queries:
http(s)://{es_endpoint}/_cat/indices/*capm*?v
http(s)://{es_endpoint}/_cat/indices/?v&s=ss:desc&h=health,store.size,pri.store.size,pri,rep,store.size,pri.store.size,docs.count,docs.deleted,index,cds
http(s)://{es_endpoint}/_cluster/health?pretty&human
Fore more details refer to : https://knowledge.broadcom.com/external/article/207215
- result of : df -h
c) from NFS server
- result of : df -h
Additional Information
Attachments
Feedback
