Workload Cluster Deployment Fails with Kubelet Error: "cni plugin not initialized"

Products

• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

• When the guest cluster creating is initiated, it gets stuck on the first control plane node. Upon SSH into that node (ssh capv@<node-ip>), it is seen that the following components are in a running state: kube-proxy, kube-vip, kube-controller-manager, etcd, kube-apiserver and kube-scheduler.


• Examining the Kubelet on the control plane node, the service is running (systemctl status kubelet.service), but the logs continuously show that the CNI is not initialized:
  

  journalctl -xeu kubelet

  "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"

Environment

TKGm 2.4.0+

Cause

• The Cluster API components (CAPI and CAPV) are unable to communicate with the new workload cluster. This can be confirmed by checking the CAPI logs: 
  
  E0101 12:00:00.000000       1 controller.go:329] "Reconciler error" err="failed to create cluster accessor: error creating http client and mapper for remote cluster \"default/workload-cluster\": error creating client for remote cluster \"default/workload-cluster\": error getting rest mapping: failed to get API group resources: unable to retrieve the complete list of server APIs: v1: Get \"https://<ip>:6443/api/v1?timeout=10s\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="default/workload-cluster-controlplane-####" namespace="default" name="workload-cluster-controlplane-####" reconcileID=""


• CAPI is responsible for provisioning and managing the lifecycle of Kubernetes clusters. If the API server of the new workload cluster cannot communicate with CAPI on the management cluster, it won't be able to complete its setup tasks - including initializing the CNI.
  
  
• A firewall rule is blocking communication on port 6443 to the workload cluster which prevents the workload cluster from registering with the management cluster. 

Resolution

Allow port 6443 in the firewall configuration for the cluster to be deployed successfully. 

Review the below documents to ensure proper networking configuration: 

• If applicable, review the proxy settings and any other networking configurations specified in the cluster configuration file during creation: Configuration File Variable Reference
• Verify the overall network setup in the environment to ensure proper connectivity between the management cluster and the workload cluster: Security and Compliance