Symptoms:
Attempting to create a snapshot save for ETCD DB is not working in TKGi.
Error:
/var/vcap/jobs/etcd/bin/etcdctl snapshot save snapshot.db --endpoints=https://master-1.etcd.cfcr.internal:2379 Error: snapshot must be requested to one selected node, not multiple [https://master-0.etcd.cfcr.internal:2379 https://master-1.etcd.cfcr.internal:2379 https://master-2.etcd.cfcr.internal:2379 https://master-1.etcd.cfcr.internal:2379]
By Default etcdctl command in TKGi is pointing to all the members as endpoint as defined in Utils.sh Script in the master nodes.
Resolution:
Steps
sudo find /var/vcap/data/ -iname utils.sh
./utils.sh
sudo /var/vcap/jobs/etcd/bin/etcdctl snapshot save /var/vcap/store/snapshot.db
See below Example:
master/94b9e8c6-7186-4379-8db3-af9c5a66151b:/var/vcap/data/jobs/etcd/2d4e497d9daff5c51dee925e42c9f9acc7758491/bin# ls
authentication-setup drain ensure-member-healthy etcd etcdctl post-start pre-start setup snapshot.db utils.sh
master/94b9e8c6-7186-4379-8db3-af9c5a66151b:/var/vcap/data/jobs/etcd/2d4e497d9daff5c51dee925e42c9f9acc7758491/bin# cat utils.sh
etcd_endpoint_address="https://master-2.etcd.cfcr.internal:2379"
etcd_endpoints="https://master-0.etcd.cfcr.internal:2379,https://master-1.etcd.cfcr.internal:2379,https://master-2.etcd.cfcr.internal:2379"
etcd_peer_address="https://master-2.etcd.cfcr.internal:2380"
etcd_peers="01615bd2-e871-495d-868c-e133d8a40054=https://master-0.etcd.cfcr.internal:2380,8d3ed3ad-fbe6-4ca9-8f2d-b055d079c504=https://master-1.etcd.cfcr.internal:2380,94b9e8c6-7186-4379-8db3-af9c5a66151b=https://master-2.etcd.cfcr.internal:2380"
**** Remove the other 2 members from etcd_endpoints *****
master/94b9e8c6-7186-4379-8db3-af9c5a66151b:/var/vcap/data/jobs/etcd/2d4e497d9daff5c51dee925e42c9f9acc7758491/bin# ./utils.sh
master/94b9e8c6-7186-4379-8db3-af9c5a66151b:/var/vcap/data/jobs/etcd/2d4e497d9daff5c51dee925e42c9f9acc7758491/bin# /var/vcap/data/jobs/etcd/2d4e497d9daff5c51dee925e42c9f9acc7758491/bin/etcdctl snapshot save snapshot.db
{"level":"info","ts":"2023-04-24T09:40:44.000Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"snapshot.db.part"}
{"level":"info","ts":"2023-04-24T09:40:44.017Z","logger":"client","caller":"[email protected]/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-04-24T09:40:44.017Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://master-2.etcd.cfcr.internal:2379"}
{"level":"info","ts":"2023-04-24T09:40:44.106Z","logger":"client","caller":"[email protected]/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-04-24T09:40:44.122Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://master-2.etcd.cfcr.internal:2379","size":"4.2 MB","took":"now"}
{"level":"info","ts":"2023-04-24T09:40:44.123Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"snapshot.db"}
Snapshot saved at snapshot.db
Users are unable to take ETCD backup by running snapshot save before running Compression and Defragmentation