TKG package reconciliation fails with kapp ownership errors and ImagePullBackOff
search cancel

TKG package reconciliation fails with kapp ownership errors and ImagePullBackOff

book

Article ID: 435709

calendar_today

Updated On:

Products

Tanzu Kubernetes Runtime VMware vSphere ESXi

Issue/Introduction

When managing VMware Tanzu Kubernetes Grid (TKG) workload clusters within vSphere with Tanzu (VKS), you may encounter the following cascading cluster failures:

  • Package Reconcile Failures: Running kubectl get pkgi -A on the cluster reveals core packages (specifically antrea, metrics-server, and secretgen-controller) in a Reconcile failed state.

  • Kapp Ownership Errors: Inspecting the failed antrea package (kubectl describe pkgi <antrea-package-name> -n vmware-system-tkg) reports the following ownership errors: kapp: Error: Ownership errors: - Resource 'role/vmware-system-antrea-register (rbac.authorization.k8s.io/v1) namespace: default' is already associated with a different label 'kapp.k14s.io/app=<OLD_APP_ID>' Resource 'rolebinding/vmware-system-antrea-register (rbac.authorization.k8s.io/v1) namespace: default' is already associated with a different label 'kapp.k14s.io/app=<OLD_APP_ID>'

  • Network Plugin Failures: Kubelet logs report: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized.
  • Node Remediation Loops: Because the Container Network Interface (CNI) fails to initialize, nodes remain in a NotReady state for longer than the configured timeout. The Cluster API (CAPI) MachineHealthCheck subsequently marks the control plane and worker nodes as UnhealthyNode and triggers continuous remediation and reprovisioning loops.

  • Control Plane Unavailability: API server logs show 503 Service Unavailable or connection refused when attempting to reach aggregated APIs (such as metrics.k8s.io or system.antrea.io), and ImagePullBackOff errors occur for core components.

Cause

This issue occurs when orphaned Role-Based Access Control (RBAC) resources (role and rolebinding for vmware-system-antrea-register in the default namespace) retain old kapp.k14s.io/app labels following a previous deployment or upgrade. The kapp-controller detects this label mismatch during reconciliation and aborts the Antrea package deployment to prevent overriding unowned resources. Consequently, the Antrea Container Network Interface (CNI) fails to initialize, preventing nodes from reaching a Ready state and triggering continuous Cluster API (CAPI) MachineHealthCheck remediation loops.

Resolution

Remove the orphaned RBAC resources manually to allow the kapp-controller to recreate them with the correct application labels during the next reconciliation loop.

  1. Change kubectl context to the affected workload cluster: 

    kubectl config use-context <workload-cluster-context-name>

  2. Delete the conflicting resources in the workload cluster's default namespace:

    kubectl delete role vmware-system-antrea-register -n default
    kubectl delete rolebinding vmware-system-antrea-register -n default

  3. Force an immediate reconcile by pausing and unpausing the package:

    kubectl patch pkgi <antrea-package-name> -n vmware-system-tkg --type merge -p '{"spec":{"paused":true}}'
    kubectl patch pkgi <antrea-package-name> -n vmware-system-tkg --type merge -p '{"spec":{"paused":false}}'

  4. Verify the packages succeed and nodes enter a Ready state:

    kubectl get pkgi -A
    kubectl get nodes


Additional Information:
For issues requiring further technical assistance, see Contact Broadcom Support.