DX OI - Unable to use the Filter option from "Alarm Analytics", it shows "Error Retrieving" message

Products

• DX Operational Intelligence

Issue/Introduction

 

Environment

DX Operational Intelligence 20.x + 

Cause

During investigation it was found that one of the elastic servers was running out of disk space impacting kafka and data processing

Resolution

1. Switch to the DXI project or namespace:

If openshift:
oc project <your-project>

If kubernetes:
kubectl config set-context --current --namespace=<your-namespace>

 

2) Stop DX Platform services 
 
cd <DX-HOME>/bin
./dx-admin.sh stop
 

3) Wait until all the pods are down. Run the following command to check the status of the pods:

kubectl get -n<your project name> pods

You can ignore the pods in "Completed" status

 

4) Increase HD in affected Elastic Server(s)

 

5) Start DX Platform
 
cd <DX-HOME>/tools
./dx-admin.sh start
 
this process can take several minutes
 

6) Wait until all the pods are up. Run the following command to check the status of the pods:

kubectl get -n<your project name> pods

 

7) Verify you can use the filter options now

 

8) (Optional) You can safely delete all the evicted pods (if any available): 
for evicted in $(kubectl get pods | grep "Evicted" | awk '{print $1}'); do kubectl delete pod ${evicted}; done

 

Additional Information

DX AIOPs Troubleshooting, Common Issues and Best Practices
https://knowledge.broadcom.com/external/article/190815