How to add a new host to an existing Patroni cluster
Article ID: 296381
Updated On:
Products
VMware Tanzu Greenplum
Issue/Introduction
This KB describes how to add a new node to and existing Patroni cluster.
It will add the new node to the existing ETCD cluster and to the Patroni cluster.
This may be used if an existing node needs to be replaced or rebuilt due to hardware issues.
It will assume that all required RPMs are installed on the host as described in VMware Postgres High Availability with Patroni.
It will add the new node to the existing ETCD cluster and to the Patroni cluster.
This may be used if an existing node needs to be replaced or rebuilt due to hardware issues.
It will assume that all required RPMs are installed on the host as described in VMware Postgres High Availability with Patroni.
Environment
Product Version: 11.5
Resolution
NOTE: The example commands in these instructions are 192.168.50.26, 192.168.50.27 and 192.168.50.28. These will need to be replaced with the correct IP addresses or hostnames of the servers in the cluster.
NOTE: All the "etcdctl" commands are preceeded by "ETCDCTL_API=3". This specifies that the API used in version 3, rather than version 2.
List the members in the cluster:
Notes:
Note: If there are errors starting the new etcd node check the logs with:
Copy the Patroni yaml configuration file from and existing host to the local host.
Modify the file to specify correct "name", "restapi"->"listen" and "restapi"->"connect_address" in the file.
NOTE: All the "etcdctl" commands are preceeded by "ETCDCTL_API=3". This specifies that the API used in version 3, rather than version 2.
List the members in the cluster:
# ETCDCTL_API=3 etcdctl --endpoints=192.168.50.26:2379,192.168.50.27:2379,192.168.50.28:2379 member list
6335e8706b01aa3f, started, etcd03, http://192.168.50.28:2380, http://192.168.50.28:2379
716f9cd53e4fd260, started, etcd01, http://192.168.50.26:2380, http://192.168.50.26:2379
773d62658cac47cf, started, etcd02, http://192.168.50.27:2380, http://192.168.50.27:2379
Check the status of the cluster:
# ETCDCTL_API=3 etcdctl --endpoints=192.168.50.26:2379,192.168.50.27:2379,192.168.50.28:2379 endpoint status Failed to get the status of endpoint 192.168.50.28:2379 (context deadline exceeded) <<< DEAD host, note the member id from the previous output 192.168.50.26:2379, 716f9cd53e4fd260, 3.3.11, 16 kB, false, 5895, 2211714 192.168.50.27:2379, 773d62658cac47cf, 3.3.11, 16 kB, true, 5895, 2211714Remove the host which is not responding:
# ETCDCTL_API=3 etcdctl --endpoints=192.168.50.26:2379,192.168.50.27:2379,192.168.50.28:2379 member remove 6335e8706b01aa3f
Member 6335e8706b01aa3f removed from cluster 7c82e7761325e1d
Copy the etcd config file from an existing/working host to the new host and change the values as appropriately:Notes:
- The line ETCD_INITIAL_CLUSTER_STATE="existing" will most likely need to be added to the configuration file.
- The ETCD_NAME should be the name of the new node
- ETCD_INITIAL_ADVERTISE_PEER_URLS and ETCD_ADVERTISE_CLIENT_URLS need to specify the IP address or hostname of the new node
- ETCD_INITIAL_CLUSTER should specify all nodes in the ETCD cluster, including the new host being added.
# cat /etc/etcd/etcd.conf ETCD_DATA_DIR="/var/lib/etcd/" ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380" ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379" ETCD_NAME="etcd03" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.50.28:2380" ETCD_ADVERTISE_CLIENT_URLS="http://192.168.50.28:2379" ETCD_INITIAL_CLUSTER="etcd01=http://192.168.50.26:2380,etcd02=http://192.168.50.27:2380,etcd03=http://192.168.50.28:2380" ETCD_INITIAL_CLUSTER_STATE="existing"Add the new node to the cluster:
# ETCDCTL_API=3 etcdctl --endpoints=192.168.50.26:2379,192.168.50.27:2379,192.168.50.28:2379 member add etcd03 --peer-urls="http://192.168.50.28:2380" Member 7b2c0c2d75acfdf6 added to cluster 7c82e7761325e1d ETCD_NAME="etcd03" ETCD_INITIAL_CLUSTER="etcd01=http://192.168.50.26:2380,etcd02=http://192.168.50.27:2380,etcd03=http://192.168.50.28:2380" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.50.28:2380" ETCD_INITIAL_CLUSTER_STATE="existing"Start etcd on the new node:
# systemctl start etcd;Check the status:
# systemctl status etcd;
● etcd.service - Etcd Server
Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2022-06-10 11:30:27 UTC; 6s ago
Main PID: 4189 (etcd)
CGroup: /system.slice/etcd.service
└─4189 /usr/bin/etcd --name=etcd03 --data-dir=/var/lib/etcd/pg.etcd --listen-client-urls=http://0.0.0.0:2379
Jun 10 11:30:27 etcd03 etcd[4189]: established a TCP streaming connection with peer 716f9cd53e4fd260 (stream MsgApp v2 reader)
Jun 10 11:30:27 etcd03 etcd[4189]: established a TCP streaming connection with peer 716f9cd53e4fd260 (stream Message reader)
Jun 10 11:30:27 etcd03 etcd[4189]: peer 773d62658cac47cf became active
Jun 10 11:30:27 etcd03 etcd[4189]: established a TCP streaming connection with peer 773d62658cac47cf (stream Message reader)
Jun 10 11:30:27 etcd03 etcd[4189]: established a TCP streaming connection with peer 773d62658cac47cf (stream MsgApp v2 reader)
Jun 10 11:30:27 etcd03 etcd[4189]: established a TCP streaming connection with peer 716f9cd53e4fd260 (stream MsgApp v2 writer)
Jun 10 11:30:27 etcd03 etcd[4189]: established a TCP streaming connection with peer 716f9cd53e4fd260 (stream Message writer)
Jun 10 11:30:27 etcd03 etcd[4189]: established a TCP streaming connection with peer 773d62658cac47cf (stream Message writer)
Jun 10 11:30:27 etcd03 etcd[4189]: established a TCP streaming connection with peer 773d62658cac47cf (stream MsgApp v2 writer)
Jun 10 11:30:27 etcd03 etcd[4189]: 7b2c0c2d75acfdf6 initialzed peer connection; fast-forwarding 8 ticks (election ticks 10) with 2 active peer(s)
List the list the members of the cluster:
# ETCDCTL_API=3 etcdctl --endpoints=192.168.50.26:2379,192.168.50.27:2379,192.168.50.28:2379 member list 716f9cd53e4fd260, started, etcd01, http://192.168.50.26:2380, http://192.168.50.26:2379 773d62658cac47cf, started, etcd02, http://192.168.50.27:2380, http://192.168.50.27:2379 7b2c0c2d75acfdf6, started, etcd03, http://192.168.50.28:2380, http://192.168.50.28:2379 <<< new memver added and shows as startedGet the status of the endpoints:
# ETCDCTL_API=3 etcdctl --endpoints=192.168.50.26:2379,192.168.50.27:2379,192.168.50.28:2379 endpoint status 192.168.50.26:2379, 716f9cd53e4fd260, 3.3.11, 16 kB, false, 5895, 2211717 192.168.50.27:2379, 773d62658cac47cf, 3.3.11, 16 kB, true, 5895, 2211717 192.168.50.28:2379, 7b2c0c2d75acfdf6, 3.3.11, 16 kB, false, 5895, 2211717 <<< Newly added node
Note: If there are errors starting the new etcd node check the logs with:
# journalctl -u etcd | tail -100Adding the host to Patroni cluster:
Copy the Patroni yaml configuration file from and existing host to the local host.
Modify the file to specify correct "name", "restapi"->"listen" and "restapi"->"connect_address" in the file.
: : name: etcd03 restapi: listen: 192.168.50.28:8008 connect_address: 192.168.50.28:8008 : :Start Patroni:
# systemctl start patroniThis should result in the a full pg_basebackup starting for the node and eventually it will sync to the running leader.
Feedback
Yes
No
Powered by
