DX OI - Unable to use the Filter option from "Alarm Analytics", it shows "Error Retrieving" message

book

Article ID: 218858

calendar_today

Updated On:

Products

DX Operational Intelligence

Issue/Introduction

 

Cause

During investigation it was found that one of the elastic servers was running out of disk space impacting kafka and data processing

Environment

DX Operational Intelligence 20.x + 

Resolution

1. Switch to the DXI project or namespace:

If openshift:
oc project <your-project>

If kubernetes:
kubectl config set-context --current --namespace=<your-namespace>

 

2) Stop DX Platform services 
 
cd <DX-HOME>/bin
./dx-admin.sh stop
 

3) Wait until all the pods are down. Run the following command to check the status of the pods:

kubectl get -n<your project name> pods

You can ignore the pods in "Completed" status

 

4) Increase HD in affected Elastic Server(s)

 

5) Start DX Platform
 
cd <DX-HOME>/tools
./dx-admin.sh start
 
this process can take several minutes
 

6) Wait until all the pods are up. Run the following command to check the status of the pods:

kubectl get -n<your project name> pods

 

7) Verify you can use the filter options now

 

8) (Optional) You can safely delete all the evicted pods (if any available): 
for evicted in $(kubectl get pods | grep "Evicted" | awk '{print $1}'); do kubectl delete pod ${evicted}; done

 

Additional Information

DX AIOPs Troubleshooting, Common Issues and Best Practices
https://knowledge.broadcom.com/external/article/190815

Attachments