DX O2 Jarvis (kafka, zookeeper, elasticSearch) Troubleshooting

Products

• DX Operational Observability

Issue/Introduction

Symptoms:

- New alarms don't appear in OI Console
- ServiceNow tickets creation is not working
- Alarms are not getting updated
- New tenants don't appear in Elastic queries, ERROR 400 and 500 in OI Service and Performance Analytics 

The following is a high-list of techniques and suggestions to employ when troubleshooting Jarvis (kafka, zookeeper, elasticsearch) common performance and configuration issues.

A) Checklist 
B) What diagnostic files should I gather for CA Support?

 

Environment

DX O2 25, 26,*

Resolution

Table of Contents

• A) Checklist
  • OVERVIEW
  • CHECK #1 : Jarvis Health
  • CHECK #2 : ElasticSearch Health
  • CHECK #3 : Zookeeper to Kafka connectivity
  • CHECK #4 : Kafka Health
  • CHECK #5 : Kron Health
• B) What to collect if the problem persist?

 

A) Checklist

 

OVERVIEW

 

CHECK #1 : Jarvis Health

 

DX AIOps - How to check Jarvis services Health 

 

CHECK #2 : ElasticSearch Health

 

DX AIOps - How to check Elastic Health 

 

 

CHECK #3 : Zookeeper to Kafka connectivity

 

DX AIOps - How to check the Zookeeper to Kafka connectivity 

 

CHECK #4 : Kafka Health

DX AIOps - How to check Kafka Health 

CHECK #5 : Kron Health

DX AIOps - How to check Jarvis Kron Health 

 

 

 

B) What to collect if the problem persist?

If after applying the above checks and recommendations the problem persist, collect the below logs and contact Broadcom Support:

 

<NFS>/jarvis/apis/logs/<jarvis-apis-pod>/*.log
<NFS>/jarvis/indexer/logs/<jarvis-indexer-pod>/*.log
<NFS>/jarvis/kafka-logs/kafka-<#>/*.log
<NFS>/jarvis/esutils/logs/<jarvis-esutils-pod>/*.log
<NFS>/jarvis/zookeeper-logs/zookeeper-<#>/*.log

 

Additional Information

AIOPs - Troubleshooting, Common Issues and Best Practices