Guest cluster nodes are in Unknown state and VM console shows "soft lockup - CPU## stuck'"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• The worker/control plane nodes are in status "Not Ready" or "Unknown".
  
  
• VM console shows the following errors:
  
  watchdog: BUG: soft lockup - CPU## stuck for 20s![containerd-###]
watchdog: BUG: soft lockup - CPU#4 stuck for 16s! [node_exporter:####]
watchdog: BUG: soft lockup - CPU#1 stuck for 12s! [kworker/1:1:####] 
watchdog: BUG: soft lockup - CPU#4 stuck for 32s! [node_exporter:####] 
watchdog: BUG: soft lockup - CPU#0 stuck for 33s! [runc:#######] 
watchdog: BUG: soft lockup - CPU#1 stuck for 31s! [containerd-shim:#######]
watchdog: BUG: soft lockup - CPU#1 stuck for 54s! [kthreadd:#######]
  
  
• kubelet on the node is not Running
  
  
• Creating a new cluster in the namespace gives following errors on the existing cluster:
  

  unable to reconcile kubeadm ConfigMap's CoreDNS info: unable to retrieve kubeadm Configmap from the guest cluster: failed to get server groups: Get "https://<IP>:6443/api?timeout=10s": dial tcp <IP>:6443: connect: connection refused

  unable to retrieve kube-proxy daemonset from the guest cluster: failed to get server groups: Get "https://<IP>:6443/api?timeout=10s": dial tcp <IP>:6443: connect: connection refused

  failed to create ClusterRoleBinding: failed to get server groups: Get "https://<IP>:6443/api?timeout=10s": dial tcp <IP>:6443: connect: connection refused

Environment

vSphere with Kubernetes

Cause

CPU exhaustion within the vSphere Namespace and resource constraints on the control plane nodes.

Resolution

Modify the resource configurations for the affected vSphere Namespace:

1. Navigate to the vSphere Namespace settings in the vCenter UI.
2. Increase the Resource Limits as per requirement.