Unable to create/update Tanzu Kubernetes Clusters (TKCs) to kubelet error: dbus: connection closed by user

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

Symptoms:

• Tanzu Kubernetes Clusters (TKCs) show as disconnected in TMC
• Creating/updating Tanzu Kubernetes Clusters (TKCs) get stuck with no VMs being created in vSphere
• When going to the vSphere Client Menu > Workload Management > Supervisor > Monitor > Events > Kubernetes, you see warnings with the following
  • Machine <machine_name> has unhealthy node
  • VM does not have an IP address
  • VM is not yet created
• When running kubectl get nodes, you see many control plane and worker nodes in a NotReady status, but there are the required number of control plane and worker nodes in a Ready state. 
• kubectl get machinedeployments -n <namespace> shows the problem cluster's Phase state as ScalingUp
• Running kubectl get machine -n <namespace> | grep <cluster_name> shows machine items without a corresponding vSphere VM
• Describing a node (kubectl describe node <node_name>) shows:
  

  Warning  FailedCreatePodContainer  28s (x4490 over 16h)  kubelet  unable to ensure pod container exists: failed to create container for [kubepods besteffort <pod_name>] : dbus: connection closed by user

• Running systemctl restart kubelet on the nodes with the error above also show "dbus: connection closed by user"

Environment

vSphere with Tanzu 7

Resolution

1. SSH to the node with the kubelet error using either method below:
   
   1. SSH to TKG Service Cluster Nodes as the System User Using a Private Key
   2. SSH to TKG Service Cluster Nodes as the System User Using a Password
2. Restart the kubelet service: systemctl restart kubelet
3. Verify that there are no errors with the kubelet service by running systemctl status kubelet
4. With kubelet now healthy, the pods that were previously unable to be scheduled should start running on the now healthy nodes
5. Remove any nodes stuck in a NotReady state by running kubectl delete node <node_name>