Delayed alarms in DX O2

Products

• DX Operational Observability
• CA Performance Management
• Spectrum

Issue/Introduction

Alarms from spectrum and netops integration are not reported on time.

Environment

• DXO2
• NetOps integration with DX O2
• Spectrum integration with DX O2

Cause

1) Incorrect configuration of NetOps connector: the below error is reported in the dxo2 jarvis-lean-indexer pod log which indicates that the encrypted cohortid in tenants.properties is incorrect  

"WARN  [pool-5-thread-3] AvroDocumentIndexer:394 - [Instance: #####-####-####-####-##########][Thread: pool-5-thread-3] topic: BRCM_VR_ao_itoa_groups_capm_1_p1 Failed to add docs to bulk processor as service object is null,due to invalid tenantId:" 

2) To confirm the LAG in Jarvis for netops and spectrum topics, use the below commands from any kafka pod

/opt/ca/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group indexer | grep BRCM_VR_ao_itoa_alarms_all_3_p1 

/opt/ca/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group indexer | grep capm

Below example of the output illustrating a LAG in the topics

Resolution

1.Go to the NetOps OIConnector server

2.Update the tenants.properties with the correct encrypted cohortid. 

For details refer to Techdocs NetOps OIConnector documentation

 

If the problem persists contact Broadcom Support and provide the below information:

1) Jarvis-lean-indexer logs

 

a) Identify the pod name:

kubectl get pods | grep jarvis-lean-indexer

jarvis-lean-indexer-<pod-name>                   1/1     Running     0   

 

b) Collect the logs for the running pod:

<NFS-dx-directory>/dxi/jarvis/indexer/jarvis-lean-indexer-<pod-name>/*

 

2) Normalized Alarm Service logs

 

a) Identify the pod name:

kubectl get pods | grep normalized-alarm

doi-normalized-alarm-<pod-name>              1/1     Running     0    

 

b) Collect the logs for the running pod:

<NFS-dx-directory>/oi/doi-normalized-alarm/logs/doi-normalized-alarm-<pod-name>/*

 

<NFS-dx-directory>/dxi/jarvis/indexer/*

 

 

2) connect to any of the kafka pods 

 

(Kubernetes) kubectl exec -ti jarvis-kafka-0 -- bash

or

(Openshift) oc rsh jarvis-kafka-0

 

and run:

/opt/ca/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group indexer | grep BRCM_VR_ao_itoa_alarms_all_3_p1

 

/opt/ca/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group indexer | grep BRCM_VR_ao_itoa_alarms_spectrum_1_p1

 

/opt/ca/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group indexer | grep capm