TKC Upgrade Stuck with Control Plane Nodes NotReady and ImagePullBackOff Due to Paused ClusterBootstrap

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

When upgrading a Tanzu Kubernetes Cluster (TKC) / Tanzu Kubernetes Grid (TKG) workload cluster from version v1.28.7 to v1.29.15, the upgrade process becomes stuck during the control plane node rollout.

Symptoms:

• Newly deployed v1.29.15 control plane nodes remain in a NotReady state.

• antrea-agent pods in the kube-system namespace remain in an Init:ImagePullBackOff or PodInitializing state:

  Init Containers:
    antrea-agent-tweaker:
      State:          Waiting
        Reason:       ImagePullBackOff
      Image:          localhost:5000/vmware.io/antrea/antrea:v1.13.3_vmware.3
  

• Running kubectl get antreaconfig -n <SUPERVISOR_NAMESPACE> on the Supervisor Cluster returns:

  No resources found in <SUPERVISOR_NAMESPACE> namespace.
  

• The Antrea PackageInstall (pkgi) on the workload cluster or Supervisor fails to reconcile with timeout errors:

  ReconcileFailed: Error: Timed out waiting after 30s for resources:
  - job/register-placeholder (batch/v1) namespace: vmware-system-antrea
  - daemonset/antrea-agent (apps/v1) namespace: kube-system

Environment

• VMware vSphere with Tanzu / Tanzu Kubernetes Grid (TKG)

• Cluster Upgrade Path: v1.28.7 to v1.29.15

• Container Network Interface (CNI): Antrea (antrea.tanzu.vmware.com)

Cause

The ClusterBootstrap custom resource for the workload cluster is paused (spec.paused: true) on the Supervisor cluster. In this state, the Tanzu Addon Controller cannot process cluster add-ons, which prevents the creation of the AntreaConfig custom resource in the Supervisor namespace and blocks the Antrea PackageInstall (pkgi) from being generated and pushed to the workload cluster.

Resolution

To unpause ClusterBootstrap and allow the CNI package reconciliation to complete, perform the following steps:

1. Log in to the Supervisor cluster using kubectl.

2. Switch context to the Supervisor namespace containing the impacted workload cluster:

   kubectl config use-context <SUPERVISOR_NAMESPACE>
   

3. Inspect the ClusterBootstrap resource configuration for the cluster:

   kubectl describe clusterbootstrap <WORKLOAD_CLUSTER_NAME> -n <SUPERVISOR_NAMESPACE>
   

   Verify under Spec if Paused is set to true.

4. Unpause the ClusterBootstrap resource by patching the spec:

   kubectl patch clusterbootstrap <WORKLOAD_CLUSTER_NAME> -n <SUPERVISOR_NAMESPACE> --type merge -p '{"spec":{"paused":false}}'
   

5. Confirm that the AntreaConfig resource is automatically generated in the Supervisor namespace:

   kubectl get antreaconfig -n <SUPERVISOR_NAMESPACE>
   

6. Check that the Antrea PackageInstall (<namespace> / <WORKLOAD_CLUSTER_NAME>-antrea) is created and reconciled:

   kubectl get pkgi -A
   

7. Verify that the new control plane nodes transition to Ready status and that the upgrade proceeds to the worker nodes:

   kubectl get nodes -o wide