VKS Clusters Stuck Ready False - etcdserver: mvcc: database space Exceeded
Article ID: 444951
Updated On:
Products
Issue/Introduction
While connected to the Supervisor cluster context, the following symptoms are observed:
- VKS clusters are stuck in Ready False state:
kubectl get clusters -A - Scaling or nodepool creation operations are stuck or not progressing for VKS clusters
- Checking the status of ETCD shows alarm:NOSPACE on all Supervisor control plane VMs:
etcdctl endpoint status -w table etcdctl --cluster=true endpoint status -w table - Viewing VMOP controller logs show the error:
Failed to update lock: etcdserver: mvcc: database space exceeded
Environment
vSphere Supervisor 9.x
VMware Cloud Foundation Automation 9.1
VKS cluster on builtin-generic-v3.7.0 clusterClass
Cause
In VMware Cloud Foundation Automation (VCFA) 9.1 and higher, Attach operations push global configuration changes to all VKS clusters in a Supervisor cluster.
Attach operations can include: Scaling up existing node-pools, adding new node-pools
The VCFA Attach operation creates Cluster scoped resources targeting all VKS clusters in the Supervisor cluster which creates spare nodes with the latest, up-to-date configuration before propagating updated configuration to the rest of the nodes.
For environments with a large number of VKS clusters using clusterClass 3.7.0, this can result in a flood of newly created resources in a short time frame and lead to ETCD database space exhaustion in the Supervisor cluster. This database exhaustion can be automatically resolved once ETCD performs its compaction successfully.
However while the ETCD database exhaustion is ongoing, the Supervisor cluster will be unhealthy and report NOSPACE alarms for ETCD.
Creating a FDS registry can cause a similar bulk update filling up the ETCD database.
By default, the ETCD database size maximum is 2GB.
Resolution
Reach out to VMware by Broadcom Technical Support and reference this KB article.
Feedback
