AIOps - Jarvis (kafka, zookeeper, elasticSearch) Troubleshooting

Products

• DX Operational Intelligence
• DX Application Performance Management
• CA App Experience Analytics

Issue/Introduction

Symptoms:

- New alarms don't appear in OI Console
- ServiceNow tickets creation is not working
- Alarms are not getting updated
- New tenants don't appear in Elastic queries, ERROR 400 and 500 in OI Service and Performance Analytics 

The following is a high-list of techniques and suggestions to employ when troubleshooting Jarvis (kafka, zookeeper, elasticsearch) common performance and configuration issues.

A) Checklist 
B) What diagnostic files should I gather for CA Support?

 

Environment

DX AIOps 2x

Resolution

Table of Contents

• A) Checklist
  • OVERVIEW
  • CHECK #1 : Jarvis Health
  • CHECK #2 : ElasticSearch Health
  • CHECK #3 : Zookeeper to Kafka connectivity
  • CHECK #4 : Kafka Health
  • CHECK #5 : Kron Health
• B) What to collect if the problem persist?

 

A) Checklist

 

OVERVIEW

 

CHECK #1 : Jarvis Health

 

https://knowledge.broadcom.com/external/article/272224 

 

CHECK #2 : ElasticSearch Health

 

https://knowledge.broadcom.com/external/article/272228 

 

 

CHECK #3 : Zookeeper to Kafka connectivity

 

https://knowledge.broadcom.com/external/article/272229 

 

CHECK #4 : Kafka Health

https://knowledge.broadcom.com/external/article/272231 

CHECK #5 : Kron Health

https://knowledge.broadcom.com/external/article/272232 

 

 

 

B) What to collect if the problem persist?

If after applying the above checks and recommendations the problem persist, collect the below logs and contact Broadcom Support:

 

<NFS>/jarvis/apis/logs/<jarvis-apis-pod>/*.log
<NFS>/jarvis/indexer/logs/<jarvis-indexer-pod>/*.log
<NFS>/jarvis/kafka-logs/kafka-<#>/*.log
<NFS>/jarvis/esutils/logs/<jarvis-esutils-pod>/*.log
<NFS>/jarvis/zookeeper-logs/zookeeper-<#>/*.log

 

Additional Information

https://knowledge.broadcom.com/external/article/190815/dx-oi-troubleshooting-common-issues-and.html