We are not seeing any new alarm in Alarms Analytics and Predictive Insights page is empty, we used to see alarms.
Root case: Disk capacity issue in one of the Elastic nodes
Troubleshooting Reference:
DX AIOps - Jarvis (kafka, zookeeper, elasticSearch) Troubleshooting
https://knowledge.broadcom.com/external/article/189119
Issues found:
1) Indexer and kron jarvis services were in red status:
Go to http(s)://<APIS_URL>/#/All/get_health
Click, "Try it out", "Execute"
2) Found that one of the Elastic nodes was running out of disk space, disk space usage was @ 98%
Check this condition using: df -h
3) ElasticSearch was in read-only mode because of the disk capacity issue
tion [type=cluster_block_exception, reason=index [jarvis_jmetrics_1.0_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]]
[16]: index [jarvis_jmetrics_1.0_1], type [_doc], id [3f1f4151-4b4a-4a04-bde1-4ae610358e81], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [jarvis_jmetrics_1.0_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]]
[17]: index [jarvis_jmetrics_1.0_1], type [_doc], id [5b8d166c-51a5-48cf-b520-d07e4c3d1bcb], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [jarvis_jmetrics_1.0_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]]
[18]: index [jarvis_jmetrics_1.0_1], type [_doc], id [efc53df9-320a-4988-9556-0160e7e26e8b], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [jarvis_jmetrics_1.0_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]]
DX Operational Intelligence 1.3.x, 20.x
DX Application Performance Management 11.x, 20.x
You have 2 options:
Option #1: Increase disk size in ElasticSearch nodes
Option #2: Reduce data retention, Delete elastic backups Or manually delete some unnecessary elastic indices, see:
DX AIOps - NFS or Elastic Nodes disk full - How to reduce data retention
OI 20.x : https://knowledge.broadcom.com/external/article/207161
OI 1.3.x : https://knowledge.broadcom.com/external/article/188786
DX OI - Troubleshooting, Common Issues and Best Practices
https://knowledge.broadcom.com/external/article/190815/dx-oi-troubleshooting-common-issues-and.html