TKC Cluster Deployment Fails with “Container runtime network not ready”

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

When deploying a Tanzu Kubernetes Cluster (TKC) in a vSphere Kubernetes(VKS) environment, the virtual machines remain stuck in the "Creating" state.
The following error is observed upon describing the machine, or you can find this error in the  kubelet logs:

# One of the control plane node
journalctl -xeu kubelet

  Container runtime network not ready
  networkReady="Network"

This issue may stem from multiple factors, including Supervisor Cluster readiness and network configuration constraints.

Environment

VMware vSphere Kubernetes Service

Cause

The cause in this scenario is a limitation in the Avi Load Balancer configuration:

• The Avi Service Engine (SE) group was set with a maximum limit of 10 SEs, which restricts scaling and prevents successful TKC cluster provisioning

• Additionally, the Supervisor Cluster or TKGS service may not be in a healthy state, contributing to deployment failure

Resolution

1. Validate Environment Health

   1. Ensure the Supervisor Cluster is in a Ready state. If not, escalate to Broadcom Support.

   2. Confirm that the TKGS service on vCenter is running.

2. Update Avi Configuration

   1. Increase the SE group limit from 10 to 100, or adjust based on customer requirements.
      (AVI UI --> Infrastructure --> Cloud Resources --> Service Engine Group --> Edit icon --> Placement --> Virtual Services per Service Engine)

3. Redeploy the TKC Cluster

   1. Retry the cluster deployment after updating the Avi SE group configuration.