We have a large number of elasticsearch nodes in the cluster that we are moving the system (kubernetes) from vmware to physical hosts. Is it possible to reduce the elasticsearch hot node count?
Release : 21.3
Removing a Node from a Elasticsearch Cluster
============================================
Please follow the below steps
***************************
Node selection - For removing a node from Elasticsearch cluster, we can have different criteria like node with least disk consumption etc.
Execute the below GET request to get the details of the nodes in the cluster
GET {ES_URL}/_cat/nodes?pretty&v&h=name,id,ip,node.role,disk.total,disk.used,disk.avail&s=disk.used:desc
Identify the node which needs to be removed from the cluster and make a note of the IP address.
Execute the below PUT request to update the cluster settings to exclude the node,
PUT {ES_URL}/_cluster/settings
{
"transient" :{
"cluster.routing.allocation.exclude._ip" : "Node_IP_Address"
}
}
Elasticsearch will start moving the existing shards out of the node that will be removed and moves it to other nodes in the cluster.
Execute the below GET request to check the progress and whether the relocation is complete or not. relocating_shards attribute should show value as 0.
GET {ES_URL}/_cluster/health?pretty
Execute the below request and make sure that the IP of the node to be removed is not listed against any shard
GET {ES_URL}/_cat/shards?pretty&v&h=index,shard,state,node,ip,prirep
Shutdown the node that we identified in step 3 for removal from the cluster.
Revert the changes done in cluster settings in Step 4 to set the exclusion rule to empty
PUT {ES_URL}/_cluster/settings
{
"transient" : {
"cluster.routing.allocation.exclude._ip" : ""
}
}
Please note:
Do not delete elasticsearchnodes created by installer.Nodes named:- jarvis-elasticsearch-2,jarvis-elasticsearch,jarvis-elasticsearch-3
You should delete the elasticsearch nodes starting from end number of elasticsearch nodes availble. for example , if we are having elastic search nodes from 1 to 30.
Please start the deletion of the nodes in order jarvis-elasticsearch-30, jarvis-elasticsearch-29,jarvis-elasticsearch-28 . This will help in scaling up of elasticsearch nodes in future using dxi-es-admin.sh.
Some Followup Steps, after doing the above steps:-(***IMP: Below steps can be done after some days of monitoring the application. And if everything is doing good, then you can perform below steps)
You may have to delete the PV/PVCs for the elasticsearch nodes you are removing. eg:- if you are deleting the elasticsearch node of 5 then you have to delete the PV and pvc of dxi-jarvis-es-hot-5. This will help in scaling up of elasticsearch nodes in future using dxi-es-admin.sh without problem.
Removal of labels from the nodes,if labels are used.Below is the command to remove the labels.
kubectl label nodes <server-name> dxi-es-node-