DX OI - Predictive Insights stop working, the page is empty

Products

• DX Operational Intelligence

Issue/Introduction

We are not seeing any new alarm in Alarms Analytics and Predictive Insights page is empty, we used to see alarms.

 

Cause

Root case:  Disk capacity issue in one of the Elastic nodes

Troubleshooting Reference:

DX AIOps - Jarvis (kafka, zookeeper, elasticSearch) Troubleshooting
https://knowledge.broadcom.com/external/article/189119

Issues found:

1) Indexer and kron jarvis services were in red status:

Go to http(s)://<APIS_URL>/#/All/get_health
Click, "Try it out", "Execute"

2) Found that one of the Elastic nodes was running out of disk space, disk space usage was @ 98%

Check this condition using: df -h

3) ElasticSearch was in read-only mode because of the disk capacity issue

tion [type=cluster_block_exception, reason=index [jarvis_jmetrics_1.0_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]]
[16]: index [jarvis_jmetrics_1.0_1], type [_doc], id [3f1f4151-4b4a-4a04-bde1-4ae610358e81], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [jarvis_jmetrics_1.0_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]]
[17]: index [jarvis_jmetrics_1.0_1], type [_doc], id [5b8d166c-51a5-48cf-b520-d07e4c3d1bcb], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [jarvis_jmetrics_1.0_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]]
[18]: index [jarvis_jmetrics_1.0_1], type [_doc], id [efc53df9-320a-4988-9556-0160e7e26e8b], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [jarvis_jmetrics_1.0_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]]

Environment

DX Operational Intelligence 1.3.x, 20.x
DX Application Performance Management 11.x, 20.x

Resolution

You have 2 options:

Option #1: Increase disk size in ElasticSearch nodes

Option #2: Reduce data retention, Delete elastic backups Or manually delete some unnecessary elastic indices, see:

DX AIOps - NFS or Elastic Nodes disk full - How to reduce data retention
OI 20.x  : https://knowledge.broadcom.com/external/article/207161
OI 1.3.x : https://knowledge.broadcom.com/external/article/188786
 

Additional Information

DX OI - Troubleshooting, Common Issues and Best Practices
https://knowledge.broadcom.com/external/article/190815/dx-oi-troubleshooting-common-issues-and.html