ALERT: Some images may not load properly within the Knowledge Base Article. If you see a broken image, please right-click and select 'Open image in a new tab'. We apologize for this inconvenience.

DX OI - If some ElasticSearch nodes are down functionality stops working - No data

book

Article ID: 226230

calendar_today

Updated On:

Products

DX Operational Intelligence

Issue/Introduction

While testing redundancy, we shutdown some Elastic Nodes and noticed that OI does not report any data.

 

 

Cause

This problem is related to defect DE517070:

Product deployments are configured to use specfic ElasticSearch Nodes instead to ElasticSearch LoadBalancer

Environment

DX Operational Intelligence 20.2.x

Resolution

1) Scale down DOI

cd <DX-Platform Installer>/tools

./dxi-component-scale.sh -n dxi -p doi DOWN

Verify that doi pods are not running

kubectl get pods -n<namespace> | grep doi


NOTE: If you find some are still running, you can terminate them using: kubectl scale --replicas=0 deployment <doi-deployment> -n <namespace>

For example:

kubectl scale --replicas=0 deployment doi-adminui doireadserver doi-normalized-alarm doi-automic-integration doi-tenantmanagement doi-nginx -n dxi

 

2) Open the doi-configmap

kubectl edit cm doi-configmap -n<namespace>

3) Replace 'jarvis-elasticsearch,jarvis-elasticsearch-2,jarvis-elasticsearch-3'  with 'jarvis-elasticsearch-lb' for below keys:

ELASTIC_HOST
ES_CLUSTER_SERVER
ES_HOST
ES_HTTP_HOSTS
ES_HTTP_URL
ES_SERVER

Expected result:

ELASTIC_HOST: 'jarvis-elasticsearch-lb'
ES_CLUSTER_SERVER: 'jarvis-elasticsearch-lb'
ES_HOST: 'jarvis-elasticsearch-lb'
ES_HTTP_HOSTS: jarvis-elasticsearch-lb:9200
ES_HTTP_URL: 'jarvis-elasticsearch-lb'
ES_SERVER: 'jarvis-elasticsearch-lb'


4) Save the configmap


5) Scale up DOI

cd <DX-Platform Installer>/tools

./dxi-component-scale.sh -n dxi -p doi UP

Verify that doi pods are not running

kubectl get pods -n<namespace> | grep doi

 

NOTE:

There should be a total of 28 pods Running and 1 in a Completed status, here is an example:

If you find some pods are not running, you can start them using: kubectl scale --replicas=1 deployment <doi-deployment> -n <namespace>

For example:

kubectl scale --replicas=1 deployment doi-adminui -n dxi

 

Additional Information

https://knowledge.broadcom.com/external/article/190815/dx-aiops-troubleshooting-common-issues-a.html

Attachments