DX Dashboards - If ElasticSearch node#1 is shutdown DX Dashboard stops working

Products

• DX Operational Intelligence
• DX Application Performance Management
• CA App Experience Analytics

Issue/Introduction

While testing redundancy, we shutdown ElasticSearch Node 1 and now are not able to run any query

We get "Query Error 502" and elastic indices are not longer available

Environment

DX Operational Intelligence 20.2.x

DX Application Performance Management 20.2.x

DX Dashboards 20.2.x

Cause

This problem is related to defect DE517030 

Product deployments are configured to use ElasticSearch Node#1 instead to ElasticSearch LoadBalancer

Resolution

1) Edit dxi-grafana-deployment and dxi-grafana-services deployments

kubectl edit deployment dxi-grafana-deployment -n<namespace>
kubectl edit deployment dxi-grafana-services -n<namespace>

For example:

kubectl edit deployment dxi-grafana-deployment -ndxi
kubectl edit deployment dxi-grafana-services -ndxi

2) Locate DXI_GRAFANA_ELASTIC_URL environment variable

3) Replace 'jarvis-elasticsearch'  with 'jarvis-elasticsearch-lb' 

Example:

4) Save the file

Existing dxi-grafana-deployment and dxi-grafana-services pods will be terminated and new ones will be created with the new changes

5) Verify: you should be able to query DX Dashboards 

Additional Information

DX AIOPs - Troubleshooting, Common Issues and Best Practices