We are not seeing any new alarm in Alarms Analytics and Predictive Insights page is empty, we used to see alarms.
DX Operational Intelligence 1.3.x, 20.x
DX Application Performance Management 11.x, 20.x
Root case: Disk capacity issue in one of the Elastic nodes
Troubleshooting Reference:
DX AIOps - Jarvis (kafka, zookeeper, elasticSearch) Troubleshooting
https://knowledge.broadcom.com/external/article/189119
Issues found:
1) Indexer and kron jarvis services were in red status:
Go to http(s)://<APIS_URL>/#/All/get_health
Click, "Try it out", "Execute"
2) Found that one of the Elastic nodes was running out of disk space, disk space usage was @ 98%
Check this condition using: df -h
3) ElasticSearch was in read-only mode because of the disk capacity issue
tion [type=cluster_block_exception, reason=index [jarvis_jmetrics_1.0_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]]
[16]: index [jarvis_jmetrics_1.0_1], type [_doc], id [3f1f4151-4b4a-4a04-bde1-4ae610358e81], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [jarvis_jmetrics_1.0_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]]
[17]: index [jarvis_jmetrics_1.0_1], type [_doc], id [5b8d166c-51a5-48cf-b520-d07e4c3d1bcb], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [jarvis_jmetrics_1.0_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]]
[18]: index [jarvis_jmetrics_1.0_1], type [_doc], id [efc53df9-320a-4988-9556-0160e7e26e8b], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [jarvis_jmetrics_1.0_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]]
You have 2 options:
Option #1: Increase disk size in ElasticSearch nodes
Option #2: Reduce data retention, Delete elastic backups Or manually delete some unnecessary elastic indices, see:
DX AIOps - NFS or Elastic Nodes disk full - How to reduce data retention
OI 20.x : https://knowledge.broadcom.com/external/article/207161
OI 1.3.x : https://knowledge.broadcom.com/external/article/188786
DX OI - Troubleshooting, Common Issues and Best Practices
https://knowledge.broadcom.com/external/article/190815/dx-oi-troubleshooting-common-issues-and.html