VKS Cluster Unhealthy, Kubectl Commands Failing due to ETCD Database Full or Exceeded

Products

VMware vSphere 7.0 with Tanzu VMware Tanzu Kubernetes Grid Service (TKGs) vSphere with Tanzu VMware vSphere Kubernetes Service

Issue/Introduction

Kubectl commands are failing in the affected vSphere Kubernetes (VKS) cluster context, also known as a guest cluster.

When connected to the Supervisor cluster context, the following symptoms are present:

Describing the affected cluster shows the below error message for the control plane nodes in the cluster:

kubectl describe cluster -n <affected cluster namespace> <affected cluster>

"Following machines are reporting unknown etcd member status"

When connected to the affected VKS cluster context, the following symptoms are present:

All kubectl commands are failing or timing out.

The logs for kube-apiserver show the following error message:

kubectl logs -n kube-system <kube-apiserver pod name>

"etcdserver: mvcc: database space exceeded"

The logs for ETCD show error messages containing the below error:
- ```
"alarm:NOSPACE"
```

When SSH to one of the affected VKS cluster's control plane nodes, the following symptoms are present:

The ETCD database for the control plane node is equal to or greater than 2.0GB which is the default maximum size:
- ```
ls -ltrh /var/lib/etcd/member/snap
```

Environment

vSphere with Tanzu 7.0

vSphere with Tanzu 8.0

This can occur on a vSphere Kubernetes cluster regardless of whether or not it is managed by Tanzu Mission Control (TMC)

Cause

ETCD's keyspace data limit has been reached or exceeded.

The default ETCD database storage size limit is 2 GB.

Once this limit is reached or exceeded, ETCD will crash.

Kube-apiserver is reliant on ETCD being healthy.

Without kube-apiserver in a healthy state, kubectl commands will fail.

Resolution

Please open a ticket to VMware by Broadcom Technical Support referencing this KB for assistance in cleaning up ETCD database and restoring it to operational state.

Once ETCD is operational again, the root cause of what is filling up ETCD database will need to be investigated.

Otherwise, this may happen again at a rate depending on how quickly the database is actively being filled.

Additional Information

If the ETCD container is running and stable, the following command can be run to retrieve a list of the top kubernetes objects filling the ETCD DB:

SSH into one of the control plane nodes for the affected VKS cluster

Establish etcdctl alias:

alias etcdctl='/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/*/fs/usr/local/bin/etcdctl --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key --cacert /etc/kubernetes/pki/etcd/ca.crt'

Query for the list of kubernetes objects in the ETCD DB by count:

etcdctl get /registry --prefix --keys-only | grep -v ^$ | awk -F '/'  '{ h[$3]++ } END {for (k in h) print h[k], k}' | sort -nr

If the above command fails, but kubectl commands are working, the below command can be used alternatively which queries for objects with a count over 100:

kubectl get --raw=/metrics | grep apiserver_storage_objects | awk '$2>100' | sort -n -k 2

You can drill down further to find the namespace(s) containing the kubernetes object(s) where KUBERNETESOBJECT needs to be replaced with one of the culprit objects:

etcdctl get /registry/KUBERNETESOBJECT --prefix=true --keys-only | grep -v ^$ | awk -F'/' {'print $4'} | sort | uniq -c

From there, the culprits for the kubernetes objects filling up the ETCD DB will need to be looked into and adjusted.

This could be pods that are repeatedly creating objects that have built-up over time or pods that are failing to run to properly clean up said created objects.

Reach out to the culprit pod's owner or the vendor for these culprit custom resources (CR).