DX OI - Predictive Insights stop working, the page is empty

Products

• DX Operational Intelligence

Issue/Introduction

We are not seeing any new alarm in Alarms Analytics and Predictive Insights page is empty, we used to see alarms.

 

Environment

DX Operational Intelligence 1.3.x, 20.x
DX Application Performance Management 11.x, 20.x

Cause

Root case:  Disk capacity issue in one of the Elastic nodes

Troubleshooting Reference:

DX AIOps - Jarvis (kafka, zookeeper, elasticSearch) Troubleshooting
https://knowledge.broadcom.com/external/article/189119

Issues found:

1) Indexer and kron jarvis services were in red status:

Go to http(s)://<APIS_URL>/#/All/get_health
Click, "Try it out", "Execute"

2) Found that one of the Elastic nodes was running out of disk space, disk space usage was @ 98%

Check this condition using: df -h

3) ElasticSearch was in read-only mode because of the disk capacity issue

tion [type=cluster_block_exception, reason=index [jarvis_jmetrics_1.0_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]]
[16]: index [jarvis_jmetrics_1.0_1], type [_doc], id [3f1f4151-4b4a-4a04-bde1-4ae610358e81], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [jarvis_jmetrics_1.0_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]]
[17]: index [jarvis_jmetrics_1.0_1], type [_doc], id [5b8d166c-51a5-48cf-b520-d07e4c3d1bcb], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [jarvis_jmetrics_1.0_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]]
[18]: index [jarvis_jmetrics_1.0_1], type [_doc], id [efc53df9-320a-4988-9556-0160e7e26e8b], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [jarvis_jmetrics_1.0_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]]

Resolution

You have 2 options:

Option #1: Increase disk size in ElasticSearch nodes

Option #2: Reduce data retention, Delete elastic backups Or manually delete some unnecessary elastic indices, see:

DX AIOps - NFS or Elastic Nodes disk full - How to reduce data retention
OI 20.x  : https://knowledge.broadcom.com/external/article/207161
OI 1.3.x : https://knowledge.broadcom.com/external/article/188786
 

Additional Information

DX OI - Troubleshooting, Common Issues and Best Practices
https://knowledge.broadcom.com/external/article/190815/dx-oi-troubleshooting-common-issues-and.html