Guest cluster deployment via the Local Consumption Interface (LCI) fails with 'best-effort-xsmall' VM Class

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Deployments using the best-effort-xsmall VM class fail.
• Guest cluster is deployed with 1 Control-Plane and 1 Worker Node.
• Checking the guest cluster status in LCI UI show it as "Not Available"
• journalctl kubelet logs show the following entries:
  
  <date><time> <guest_cluster_name> kubelet[2359]:  2359 generic.go:455] "PLEG: Write status" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" pod="vmware-system-csi/vsphere-csi-controller-#########"

<date><time> <guest_cluster_name> kubelet[2359]:  2359 generic.go:254] "GenericPLEG: Unable to retrieve pods" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" 

<date><time><guest_cluster_name> kubelet[2359]:  2359 kubelet.go:2530] "Skipping pod synchronization" err="[container runtime is down, PLEG is not healthy: pleg was last seen active 12m52.54364169s ago; threshold is 3m0s]"

• Guest Cluster Deployments work for best-effort Medium VM class and higher.

Environment

vSphere Kubernetes Service

Cause

• The CSI controller requires a baseline of compute resources to operate. When placed on a best-effort-xsmall node alongside core Kubernetes services (such as etcd and the API server), the CSI controller exhausts the single available CPU. 
• This CPU starvation causes the container runtime to hang, leading to the [container runtime is down] error and preventing the cluster from finishing its bootstrap process.

Resolution

Broadcom Engineering is aware of this issue and currently there is no resolution.

Workaround:

• Deploy clusters using guaranteed type VM classes or the default best-effort-Medium class.
• To avoid overcommitting resources, production workloads should strictly use the guaranteed class type. Do not use small or xsmall class sizes for any worker node where workloads are being deployed in any environment (development, test, or production) to avoid running out of memory. Reference: Using VM Classes with TKG Service Clusters
•  While adding a persistent volume class is optional in LCI, this option is only recommended for VM classes with higher resource allocations.

Additional Information

Deploying a Stand-Alone VM in vSphere Supervisor Environment