Delayed alarms in DXO2

Products

• DX Operational Observability
• CA Performance Management
• Spectrum

Issue/Introduction

Alarms from spectrum and netops integration are not reported on time.

Environment

• DXO2
• NetOps and Spectrum integrations

Cause

1) Incorrect configuration of NetOps connector: the below error is reported in the dxo2 jarvis-lean-indexer pod log which indicates that the encrypted cohortid in tenants.properties is incorrect  

"WARN  [pool-5-thread-3] AvroDocumentIndexer:394 - [Instance: #####-####-####-####-##########][Thread: pool-5-thread-3] topic: BRCM_VR_ao_itoa_groups_capm_1_p1 Failed to add docs to bulk processor as service object is null,due to invalid tenantId:" 

2) To confirm the LAG in Jarvis for netops and spectrum topics, use the below commands from any kafka pod

/opt/ca/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group indexer | grep BRCM_VR_ao_itoa_alarms_all_3_p1 

/opt/ca/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group indexer | grep capm

Below example of the output illustrating a LAG in the topics

Resolution

1.Go to the NetOps OIConnector server

2.Update the tenants.properties with the correct encrypted cohortid. For details refer to Techdocs NetOps OIConnector documentation