How to removing elasticsearch from cluster

Products

• DX Operational Intelligence

Issue/Introduction

We have a large number of elasticsearch nodes in the cluster that we are moving the system (kubernetes) from vmware to physical hosts. Is it possible to reduce the elasticsearch hot node count? 

 

Environment

Release : 21.3

Resolution

Removing a Node from a Elasticsearch Cluster

============================================

Please follow the below  steps
***************************
Node selection - For removing a node from Elasticsearch cluster, we can have different criteria like node with least disk consumption etc.

Execute the below GET request to get the details of the nodes in the cluster 

GET {ES_URL}/_cat/nodes?pretty&v&h=name,id,ip,node.role,disk.total,disk.used,disk.avail&s=disk.used:desc

Identify the node which needs to be removed from the cluster and make a note of the IP address.

Execute the below PUT request to update the cluster settings to exclude the node, 

PUT {ES_URL}/_cluster/settings
{
"transient" :{
                   "cluster.routing.allocation.exclude._ip" : "Node_IP_Address"
                   }
}

Elasticsearch will start moving the existing shards out of the node that will be removed and moves it to other nodes in the cluster.

Execute the below GET request to check the progress and whether the relocation is complete or not. relocating_shards attribute should show value as 0.

GET {ES_URL}/_cluster/health?pretty

Execute the below request and make sure that the IP of the node to be removed is not listed against any shard

GET {ES_URL}/_cat/shards?pretty&v&h=index,shard,state,node,ip,prirep

Shutdown the node that we identified in step 3 for removal from the cluster.

Revert the changes done in cluster settings in Step 4 to set the exclusion rule to empty

PUT {ES_URL}/_cluster/settings
{
"transient" : {
                     "cluster.routing.allocation.exclude._ip" : ""
                    }
}

Please note:

Do not delete elasticsearchnodes created by installer.Nodes named:- jarvis-elasticsearch-2,jarvis-elasticsearch,jarvis-elasticsearch-3

You should delete the elasticsearch nodes starting from end number of elasticsearch nodes availble. for example , if we are having elastic search nodes from 1 to 30.

Please start the deletion of the nodes in order jarvis-elasticsearch-30, jarvis-elasticsearch-29,jarvis-elasticsearch-28 .  This will help in scaling up of elasticsearch nodes in future using  dxi-es-admin.sh.

Some Followup Steps, after doing the above steps:-(***IMP: Below steps can be done after some days of monitoring the application.  And if everything is doing good, then you can perform below steps)

You may have to delete the PV/PVCs for the elasticsearch nodes you are removing. eg:- if you are deleting the elasticsearch node of 5 then you have to delete the PV and pvc of  dxi-jarvis-es-hot-5. This will help in scaling up of elasticsearch nodes in future using  dxi-es-admin.sh  without problem.

Removal of labels from the nodes,if labels are used.Below is the command to remove the labels.

kubectl label nodes <server-name> dxi-es-node-