VKS clusters report unhealthy and nodes stuck in NotReady state due to resource contention at namespace level

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• VKS clusters are showing as unhealthy.
• You will observe that the nodes are in a NotReady state, and there are associated ETCD issues present within the cluster.
• kube-apiserver to continuously restart, indicating ETCD re-initializing errors:

E#### HH:MM:SS       1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:\"etcdserver: rpc not supported for learner\"}: etcdserver: rpc not supported for learner" logger="UnhandledError"
E#### HH:MM:SS       1 storage_rbac.go:187] "Unhandled Error" err="unable to initialize clusterroles: etcdserver: rpc not supported for learner" logger="UnhandledError"
W#### HH:MM:SS       1 storage_scheduling.go:106] unable to get PriorityClass system-node-critical: etcdserver: rpc not supported for learner. Retrying...
W#### HH:MM:SS       1 reflector.go:569] storage/cacher.go:/resourcequotas: failed to list *core.ResourceQuota: etcdserver: rpc not supported for learner
E#### HH:MM:SS       1 cacher.go:478] cacher (resourcequotas): unexpected ListAndWatch error: failed to list *core.ResourceQuota: etcdserver: rpc not supported for learner; reinitializing...

• ETCD database is failing health checks with high latency (190ms - 285ms) despite physical storage latency being under 3ms.
• %CSTP (Co-Stop) value for the guest cluster VM's show extremely high  in "esxtop" results at the ESXi level 

Environment

• VMware Cloud Foundation (VCF) 9.x
• vSphere Supervisor
• VKS cluster

Cause

This issue is caused by insufficient namespace resources allocated for the nodes within the clusters. The contention is the direct result of the VCF Automation namespace resource pool having restrictive limits set while the underlying VM classes were oversubscribed and configured as best-effort rather than guaranteed.

Resolution

To resolve this issue, you must increase the namespace resource limits from VCF Automation:

• For VCF Automation 9.1: VCF Automation 9.1 includes native UI support for editing namespace limits. Navigate to the UI and edit the Namespace Settings to adjust the resource limits. Follow Edit Namespace Settings in VCF Automation 9.1

• For VCF Automation 9.0: Follow the detailed instructions provided in Knowledge Base Article 449837: Unable to modify Namespace resource limits via the VCF Automation 9.0 User Interface

Additional Information

Best practice is to plan for unlimited resources for namespaces. If you must limit resources, ensure you set the limits according to the specific deployed load. Configure a vSphere Namespace for TKG Service Clusters

Always align the appropriate VM class to ensure sufficient capacity. Using VM Classes with VKS Clusters