search cancel

DX OI - If some ElasticSearch nodes are down functionality stops working - No data

book

Article ID: 226230

calendar_today

Updated On:

Products

DX Operational Intelligence

Issue/Introduction

While testing redundancy, we shutdown some Elastic Nodes and noticed that OI does not report any data.

 

 

Environment

DX Operational Intelligence 20.2.x

Cause

This problem is related to defect DE517070:

Product deployments are configured to use specfic ElasticSearch Nodes instead to ElasticSearch LoadBalancer

Resolution

1) Scale down DOI

cd <DX-Platform Installer>/tools

./dxi-component-scale.sh -n dxi -p doi DOWN

Verify that doi pods are not running

kubectl get pods -n<namespace> | grep doi


NOTE: If you find some are still running, you can terminate them using: kubectl scale --replicas=0 deployment <doi-deployment> -n <namespace>

For example:

kubectl scale --replicas=0 deployment doi-adminui doireadserver doi-normalized-alarm doi-automic-integration doi-tenantmanagement doi-nginx -n dxi

 

2) Open the doi-configmap

kubectl edit cm doi-configmap -n<namespace>

3) Replace 'jarvis-elasticsearch,jarvis-elasticsearch-2,jarvis-elasticsearch-3'  with 'jarvis-elasticsearch-lb' for below keys:

ELASTIC_HOST
ES_CLUSTER_SERVER
ES_HOST
ES_HTTP_HOSTS
ES_HTTP_URL
ES_SERVER

Expected result:

ELASTIC_HOST: 'jarvis-elasticsearch-lb'
ES_CLUSTER_SERVER: 'jarvis-elasticsearch-lb'
ES_HOST: 'jarvis-elasticsearch-lb'
ES_HTTP_HOSTS: jarvis-elasticsearch-lb:9200
ES_HTTP_URL: 'jarvis-elasticsearch-lb'
ES_SERVER: 'jarvis-elasticsearch-lb'


4) Save the configmap


5) Scale up DOI

cd <DX-Platform Installer>/tools

./dxi-component-scale.sh -n dxi -p doi UP

Verify that doi pods are not running

kubectl get pods -n<namespace> | grep doi

 

NOTE:

There should be a total of 28 pods Running and 1 in a Completed status, here is an example:

If you find some pods are not running, you can start them using: kubectl scale --replicas=1 deployment <doi-deployment> -n <namespace>

For example:

kubectl scale --replicas=1 deployment doi-adminui -n dxi

 

Additional Information

https://knowledge.broadcom.com/external/article/190815/dx-aiops-troubleshooting-common-issues-a.html

Attachments