etcdserver: mvcc: database space exceeded TKGi

Products

VMware

Issue/Introduction

Symptoms:

Applying following article ( https://community.pivotal.io/s/article/Error-message-etcdserver-mvcc-database-space-exceeded-when-creating-resource-in-the-K8s-cluster?language=en_US) to get snapshot save for ETCD DB is not working in TKGi be default

Error:

/var/vcap/jobs/etcd/bin/etcdctl snapshot save snapshot.db --endpoints=https://master-1.etcd.cfcr.internal:2379

Error: snapshot must be requested to one selected node, not multiple [https://master-0.etcd.cfcr.internal:2379 https://master-1.etcd.cfcr.internal:2379 https://master-2.etcd.cfcr.internal:2379 https://master-1.etcd.cfcr.internal:2379]

Environment

VMware Tanzu Kubernetes Grid Integrated Edition 1.x

Cause

By Default etcdctl command in TKGi is pointing to all the members as endpoint as defined in Utils.sh Script in the master nodes.

Resolution

Resolution:

By default snapshot save as by default snapshot save is mapping to the three etcd members
Change the script utils.sh to map to one member as per steps below

Steps

Login to one of the master nodes.
Check the path of etcdctl .
There is a script called utils.sh where the the members are in endpoints by default.
```
sudo find /var/vcap/data/ -iname utils.sh
```
Do a VIM on the script and remove the two other endpoints except the one you are inside.
Run the script
```
./utils.sh
```

Try to run the command again for snapshot save .

sudo /var/vcap/jobs/etcd/bin/etcdctl snapshot save /var/vcap/store/snapshot.db

Revert back the changes in utils.sh
Follow normally below KB to compact and defrag etcd: https://community.pivotal.io/s/article/Error-message-etcdserver-mvcc-database-space-exceeded-when-creating-resource-in-the-K8s-cluster?language=en_US

See below Example:

master/94b9e8c6-7186-4379-8db3-af9c5a66151b:/var/vcap/data/jobs/etcd/2d4e497d9daff5c51dee925e42c9f9acc7758491/bin# ls

authentication-setup drain ensure-member-healthy etcd etcdctl post-start pre-start setup snapshot.db utils.sh

master/94b9e8c6-7186-4379-8db3-af9c5a66151b:/var/vcap/data/jobs/etcd/2d4e497d9daff5c51dee925e42c9f9acc7758491/bin# cat utils.sh

etcd_endpoint_address="https://master-2.etcd.cfcr.internal:2379"
etcd_endpoints="https://master-0.etcd.cfcr.internal:2379,https://master-1.etcd.cfcr.internal:2379,https://master-2.etcd.cfcr.internal:2379"
etcd_peer_address="https://master-2.etcd.cfcr.internal:2380"
etcd_peers="01615bd2-e871-495d-868c-e133d8a40054=https://master-0.etcd.cfcr.internal:2380,8d3ed3ad-fbe6-4ca9-8f2d-b055d079c504=https://master-1.etcd.cfcr.internal:2380,94b9e8c6-7186-4379-8db3-af9c5a66151b=https://master-2.etcd.cfcr.internal:2380"

**** Remove the other 2 members from etcd_endpoints *****
master/94b9e8c6-7186-4379-8db3-af9c5a66151b:/var/vcap/data/jobs/etcd/2d4e497d9daff5c51dee925e42c9f9acc7758491/bin# ./utils.sh

master/94b9e8c6-7186-4379-8db3-af9c5a66151b:/var/vcap/data/jobs/etcd/2d4e497d9daff5c51dee925e42c9f9acc7758491/bin# /var/vcap/data/jobs/etcd/2d4e497d9daff5c51dee925e42c9f9acc7758491/bin/etcdctl snapshot save snapshot.db

{"level":"info","ts":"2023-04-24T09:40:44.000Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"snapshot.db.part"}

{"level":"info","ts":"2023-04-24T09:40:44.017Z","logger":"client","caller":"[email protected]/maintenance.go:212","msg":"opened snapshot stream; downloading"}

{"level":"info","ts":"2023-04-24T09:40:44.017Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://master-2.etcd.cfcr.internal:2379"}

{"level":"info","ts":"2023-04-24T09:40:44.106Z","logger":"client","caller":"[email protected]/maintenance.go:220","msg":"completed snapshot read; closing"}

{"level":"info","ts":"2023-04-24T09:40:44.122Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://master-2.etcd.cfcr.internal:2379","size":"4.2 MB","took":"now"}

{"level":"info","ts":"2023-04-24T09:40:44.123Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"snapshot.db"}

Snapshot saved at snapshot.db

Additional Information

Impact/Risks:

Users are unable to take ETCD backup by running snapshot save before running Compression and Defragmentation