Manually defrag etcd keyspace history when auto compaction fails due to ETCD database space exhaustion

Products

• VMware

Issue/Introduction

This KB will help to defragmentation the etcd database size. Defragmentation operation removes the free space holes from the storage.


Symptoms:

• kube-api-server logs would contain "etcdserver: mvcc: database space exceeded" which indicates storage space exhaustion.

• When you run etcdctl command from control plane VM to get endpoint status (etcdctl endpoint status -w ) it would report "alarm:NOSPACE" in the result.

Environment

VMware Tanzu Kubernetes Grid Plus 1.x

Cause

Keyspace data exceeded. The default storage size limit is 2 GB.

Resolution

To resolve this issue, run the etcdctl commands manually from the control plane VM and defragment an etcd member.

      etcdctl defrag

You can also run defragment operations for all endpoints in the cluster associated with the default endpoint.

      etcdctl defrag --cluster

To disarm the alarm. 

            etcdctl alarm disarm

Additional Information

Since history should be periodically compacted to avoid performance degradation and eventual storage space exhaustion.The keyspace can be defragmented so it can be manually done using etcdctl. You can perform the defragmentation which helps releasing this storage space back to the file system.

Impact/Risks:
kubectl command will not work and it will show "no route to host" also it will cause etcd container to restart frequently in the cluster.