ALERT: Some images may not load properly within the Knowledge Base Article. If you see a broken image, please right-click and select 'Open image in a new tab'. We apologize for this inconvenience.

DX OI integration with NetOps PM and NFA - OI Connector 1.5 Troubleshooting

book

Article ID: 226883

calendar_today

Updated On:

Products

DX Operational Intelligence

Issue/Introduction

The following is a list of techniques and suggestions to employ when troubleshooting OI connector issues

Environment

DX NetOps OI Connector 1.5 ONLY

NOTE: OIConnector 1.5 is supported with DX NetOps Performance Management 20.2.3 or lower

Resolution

APM Gateway  Hostname

If OI 20.2.x:

- If Openshift    : oc get routes -ndxi| grep apmservices-gateway
- If Kubernetes : kubectl get ingress -ndxi | grep apmservices-gateway

- If OI SaaS or 21.x:
Go to Settings> Connector Parameters > TAS Endpoint

APM Gateway  Security Token

If OI 20.2.x:

Option 1) Login to the Tenant using and Admin account, go to DX Application Performance Management (APM)> Settings> Security > Generate New Token > select “Tenant” or "Agent" Token
Option 2) If APM is not installed, login as MASTERADMIN, go to tenants, locate your tenant, "Create a Tenant Token" 

If OI SaaS or 21.x
Go to Settings > Connector Parameters > Generate Ingestion Token

Jarvis Server  Hostname

If OI 20.2.x):

- If Openshift    : oc get routes -ndxi| grep nginx
- If Kubernetes : kubectl get ingress -ndxi | grep nginx

If OI SaaS or 21.x:
Go to Settings> Connector Parameters > Jarvis Endpoint

Tenant ID

If OI 20.2.x:

a) Obtain the "ElasticSearch endpoint"

- If Openshift:     oc get routes -n<dxi-namespace> | grep jarvis
- If Kubernetes:  kubectl get ingress -n<dxi-namespace> | grep jarvis

b) Open your browser, list all available tenants:
http://<elastic-endpoint>/ao_dxi_tenants_1_1/_search?size=200&pretty

If OI SaaS or 21.x
Go to Settings> Connector Parameters > Cohort ID

 

CHECK#2 : Check services

a) Check that the OI Connector services are up and running

service caperfcenter_oiconnector status
service caperfcenter_oiagent status

b) Check that the OI Connector services in NetOps console

If OI SaaS or 21.x:

Go to Performance Center >  Administration > System Status page
Locate the "OI Connector" section
Verify Status = Normal

If OI 20.2.x:

the OI Connector will report the status as failed. You can ignore the status.

 

CHECK#3 : Check logs 

a) Review OIConnector logs : <OIConnector-HOME>/logs

- OIConnector.log : main log file
- OIAgent*.log : NFA, ADA data collection activity


b) Enable DEBUG logging:

- OIConnector logging: <OIConnector-HOME>/conf/log4j.xml

Open ./conf/log4j.xml, change logging level from INFO to DEBUG as below:

...
<!-- ***** Root Logger definition ***** -->
    <root>
        <level value="DEBUG"/>
        <appender-ref ref="console"/>
        <appender-ref ref="complete" />
    </root>


- OI Agent service logging: <OIConnector-HOME>/conf/agent-wrapper.conf

Uncomment the below line:

#wrapper.app.parameter.2=-Ssupport

 

You need to restart the oi OI Agent service:

service caperfcenter_oiagent restart

 

c) Example of common errors or exceptions :


USE-CASE#1 :Problem with Data Aggregator impacting metrics gathering

ERROR [2020-09-12 22:58:41,943] [pool-2-thread-6] [OpenAPIInventoryQueryInfHelper] - [EVENT UNSPECIFIED Anonymous:[email protected] -> /com.ca.im.oinet.connector.sources.da.inventory.OpenAPIInventoryQueryInfHelper] OpenAPI Inventory Query Failed with Status: 500
..

INFO  [2020-09-11 20:34:33,773] [pool-7-thread-5] [StatusHolder] - [EVENT UNSPECIFIED Anonymous:[email protected] -> /com.ca.im.oinet.connector.status.StatusHolder] Updating OI_CONNECTOR_NAME status information, STATUS [UP -> UP] HEALTH [NORMAL -> FAILED] Unable to contact OpenAPI for data

Or Performance Center >  Administration > System Status page:  "Unable to connect OpenAPI for data"

Recommendation:

Restart Data Aggregator (DA), see https://knowledge.broadcom.com/external/article?articleId=190659


USE-CASE #2 : Problem with Jarvis(nginx) and/or apmservices-gateway endpoints

INFO  [2020-09-11 20:00:32,988] [pool-2-thread-10] [StatusHolder] - [EVENT UNSPECIFIED Anonymous:[email protected] -> /com.ca.im.oinet.connector.status.StatusHolder] Updating OI_CONNECTOR_NAME status information, STATUS [UP -> UP] HEALTH [NORMAL -> FAILED] Unable to contact OI Platform to send data to
ERROR [2020-09-11 20:00:32,988] [pool-2-thread-10] [RemoteDataConnectionImpl] - [EVENT UNSPECIFIED Anonymous:[email protected] -> /com.ca.im.oinet.connector.sources.RemoteDataConnectionImpl]Error posting documents to Jarvis index(itoa_groups_capm): 503_
ERROR [2020-09-11 20:00:32,989] [pool-2-thread-10] [GroupTaskImpl] - [EVENT UNSPECIFIED Anonymous:[email protected] -> /com.ca.im.oinet.connector.task.group.GroupTaskImpl] Unable to push 27 groups to data sink.
ERROR [2020-09-11 20:00:33,607] [pool-2-thread-3] [TASGroupTask] - [EVENT UNSPECIFIED Anonymous:[email protected] -> /com.ca.im.oinet.connector.task.group.TASGroupTask] Failed ingesting groups to TAS for CAPC tenant id : Coke_test Error: 503

Recommendation:

Verify that the nginx and apm-gateway endpoints are correct and are available

USE-CASE #3 : CAPM user password expired, changed or not valid.

ERROR [2020-11-27 12:42:12,213] [WrapperSimpleAppMain] [OIIntegration] - [EVENT UNSPECIFIED Anonymous:[email protected] -> /com.ca.im.oinet.connector.OIIntegration] No response from webservice - unable to configure data sources
WARN  [2020-11-27 12:42:12,230] [WrapperSimpleAppMain] [OIIntegration] - [EVENT UNSPECIFIED Anonymous:[email protected] -> /com.ca.im.oinet.connector.OIIntegration] Unable to determine CA Performance Center version

Recommendation

Update the <OIConnector-HOME>/conf/config.xml with the new encoded password, see: https://knowledge.broadcom.com/external/article/204144/dx-oi-oiconnector-not-connecting-when-c.html


d) Search for common keywords: "CLIENT_SUMMARY_NASS" , “Successfully", "Started", "JARVIS_INGEST_RECORD_COUNT"

Below are some examples:

...
INFO  [2020-09-13 09:24:50,587] [pool-2-thread-4] [InventoryTaskImpl] - [EVENT UNSPECIFIED Anonymous:[email protected] -> /com.ca.im.oinet.connector.task.inventory.InventoryTaskImpl] Successfully ingested inventory to TAS for CAPC tenant id : Coke_test

..
INFO  [2020-09-13 09:18:14,105] [pool-2-thread-2] [RemoteDataConnectionImpl] - [EVENT UNSPECIFIED Anonymous:[email protected] -> /com.ca.im.oinet.connector.sources.RemoteDataConnectionImpl] JARVIS_INGEST_RECORD_COUNT : 27
INFO  [2020-09-13 09:18:15,190] [pool-2-thread-7] [TASGroupTask] - [EVENT UNSPECIFIED Anonymous:[email protected] -> /com.ca.im.oinet.connector.task.group.TASGroupTask] Successfully ingested groups to TAS for CAPC tenant id : Coke_test

..
INFO  [2020-09-13 09:11:31,481] [pool-3-thread-1] [PersistentRegistrationCache] - Successfully loaded760 metric registrations from /opt/CA/OIConnector/conf/MetricRegistrationCache-F1B889C8-4BB8-4860-BB22-447D0EEA56B0.ser
INFO  [2020-09-13 09:11:31,492] [pool-3-thread-1] [NASSClient] - Started NASS Client.

..
INFO  [2020-09-13 09:20:35,748] [WrapperSimpleAppMain] [ServerConnector] - Started ServerConnector@76f3fc3b{HTTP/1.1,[http/1.1]}{0.0.0.0:8782}

 

CHECK#4 : Check Alarms, Metrics and Topology data from DX OI UI

a) Metrics (NASS)

Go to Performance:

In SaaS, 21.x

In 20.2.x on premise:




b) Inventory and Topology (TAS)

Go to DX OI > Services > Create a new Service

From Add Elements,, select Network > Device Names, you should be able to see your NetOps devices, below an example:

 

CHECK#5 : Check the Alarms, Metrics, Topology data using Elastic and TAS/NAS REST APIs

** This section is valid for DX On Premise 20.x, 21.x versions only, if you are using DX OI SaaS, contact Broadcom Support for assistance **


a) Alarms(ElasticSearch)


For details how to query elasticsearch refer to : https://knowledge.broadcom.com/external/article/207215


1) List all the UIM product indices:

http://es.<servername>/_cat/indices/*capm*?v

For example:

http://es.munqa001493.bpc.broadcom.net/_cat/indices/*capm*?v

Check that doc.count and size columns values increases over the time.


2) Check the content of a specific index:

http://es.<severname>/<index-name>/_search?pretty&[email protected]:desc&size=500

For example:

http://es.munqa001493.bpc.broadcom.net/ao_itoa_groups_capm_1_1/_search?pretty&[email protected]:desc&size=500


You can use https://www.epochconverter.com/ to convert values from @timestamp field to human-readable format, 

 

b) Inventory and Topology (TAS)

Option 1: Use DX Dashboard > AIOps Inventory source, see:  https://knowledge.broadcom.com/external/article/226599

 

Option 2: User REST APIs:

Open Postman (you can download postman from https://www.postman.com/downloads/)

POST API End Point to check TAS data for UIM inventory: 

http://<APMServices Gateway Host>/tas/graph/query

For example:

http://apmservices-gateway.munqa001493.bpc.broadcom.net/tas/graph/query

Headers:

Content-Type: application/json

Authorization: Bearer <Tenant Token>

Body:

  {
   "filter": {
       "op": "JOIN",
       "input": {
           "op": "AND",
           "input": [
               {
                   "op": "ATTRIBUTE",
                   "expressions": [
                       {
                           "name": "Product",
                           "values": [
                               "CAPC"
                           ]
                       }
                   ]
               }
           ]
       }
   },
   "universe": null,
   "version": null,
   "time": 0,
   "stitchingEnabled": true,
   "includeStatus": true
}

Expected Result: you  should see all new vertices added to TAS

c) Metrics(NASS)

Option 1: Use DX Dashboard > AIOps Metadata source

 

Option 2: User REST APIs:

Open Postman (you can download postman from https://www.postman.com/downloads/)

POST API End Point to check NASS Metric Metadata matching a pattern

http://<APM Service Gateway Host>/metadata/queryMetric

For example:

http://apmservices-gateway.munqa001493.bpc.broadcom.net/metadata/queryMetric

Headers:

Content-Type: application/json

Authorization: Bearer <Tenant Token>

Body:

{
   "size": 10000,
 "specifier": {
   "op": "SPEC",
   "sourceNameSpecifier": {
     "op": "REGEX",
     "pattern": "(.*)NetOps\\|CAPM(.*)|(.*)NetOps\\|ADA(.*)|(.*)NetOps\\|NFA(.*)"
   },
   "attributeNameSpecifier": {
     "op": "ALL"
   }
 }
}

Expected Result: you should see all new vertices added to NAS

CHECK#6 : Verify Jarvis, Elastic, Zookeeper and Kafka

** This section is valid for DX On Premise 20.x, 21.x versions only, if you are using DX OI SaaS, contact Broadcom Support for assistance **

AIOps - Jarvis (kafka, zookeeper, elasticSearch) Troubleshooting



C) WHAT FILES SHOULD I COLLECT FOR BROADCOM SUPPORT?

If you still need assistance, contact Broadcom Support (https://support.broadcom.com/) and provide the below information:

a) DEBUG oi_connector logs

<OIConnector>/logs/*
<OIConnector>/conf/config.xml

b) services status:

service caperfcenter_oiconnector status
service caperfcenter_oiagent status


If you are using DX OI On Premise (20.x, 21.x)

a) cluster and pods status:

kubectl get pods -n<namespace>
kubectl describe nodes -n<namespace>
kubectl get events -n<namespace>

b) from ElasticSearch

- collect result of below queries:

http(s)://{es_endpoint}/_cat/indices/*capm*?v
http(s)://{es_endpoint}/_cat/indices/?v&s=ss:desc&h=health,store.size,pri.store.size,pri,rep,store.size,pri.store.size,docs.count,docs.deleted,index,cds
http(s)://{es_endpoint}/_cluster/health?pretty&human

Fore more details refer to : https://knowledge.broadcom.com/external/article/207215

- result of : df -h

c) from NFS server

- result of : df -h



Additional Information

https://knowledge.broadcom.com/external/article/190815/aiops-troubleshooting-common-issues-and.html

Youtube DX Operational Intelligence Integrations with NetOps

 

Attachments