DX AIOps - How to check Jarvis Kron Health

Products

• DX Operational Intelligence
• DX Application Performance Management
• CA App Experience Analytics

Issue/Introduction

The following is a high-list of techniques and suggestions to employ when troubleshooting Jarvis  common performance and configuration issues.

Environment

DX AIOps 2x

Resolution

Table of Contents

• 1.Checklist
  • • USE CASE #1 : Corrupted or BIG incides affecting the kron and purging or cleaning process causing a disk full issue
    • USE CASE #2 : Nodes may be having some intermittent connectivity or disk pressure issues
• 2.What to collect if the problem persist?

1.Checklist

USE CASE #1 : Corrupted or BIG incides affecting the kron and purging or cleaning process causing a disk full issue

1) Review the kron logs, search for possible errors and exceptions

2) Restart Kron and esutils in that order 

3) If an index is greater than 100 GB, rollover is the problem.

Suggestion:

 

To check this condition: you can run:

http(s)://{es_endpoint}/_cat/indices/*?s=index,cds&h=index,ss,cds

OR

http(s)://{es_endpoint}/_cat/indices/?v&s=ss:desc&h=health,store.size,pri.store.size,pri,rep,store.size,pri.store.size,docs.count,docs.deleted,index,cds

 

Example:

 

Action to take: 

- delete the problematic index

- restart Kron and esutils pods

 

USE CASE #2 : Nodes may be having some intermittent connectivity or disk pressure issues

 

https://knowledge.broadcom.com/external/article/236940 

 

 

2.What to collect if the problem persist?

If after applying the above checks and recommendations the problem persist, collect the below logs and contact Broadcom Support:

 

<NFS>/jarvis/apis/logs/<jarvis-apis-pod>/*.log
<NFS>/jarvis/indexer/logs/<jarvis-indexer-pod>/*.log
<NFS>/jarvis/kafka-logs/kafka-<#>/*.log
<NFS>/jarvis/esutils/logs/<jarvis-esutils-pod>/*.log
<NFS>/jarvis/zookeeper-logs/zookeeper-<#>/*.log

Additional Information

https://knowledge.broadcom.com/external/article/190815/aiops-troubleshooting-common-issues-and.html