When managing VMware Tanzu Kubernetes Grid (TKG) workload clusters within vSphere with Tanzu (VKS), you may encounter the following cascading cluster failures:
kubectl get pkgi -A on the cluster reveals core packages (specifically antrea, metrics-server, and secretgen-controller) in a Reconcile failed state.kubectl describe pkgi <antrea-package-name> -n vmware-system-tkg) reports the following ownership errors: kapp: Error: Ownership errors: - Resource 'role/vmware-system-antrea-register (rbac.authorization.k8s.io/v1) namespace: default' is already associated with a different label 'kapp.k14s.io/app=<OLD_APP_ID>' Resource 'rolebinding/vmware-system-antrea-register (rbac.authorization.k8s.io/v1) namespace: default' is already associated with a different label 'kapp.k14s.io/app=<OLD_APP_ID>'NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized.NotReady state for longer than the configured timeout. The Cluster API (CAPI) MachineHealthCheck subsequently marks the control plane and worker nodes as UnhealthyNode and triggers continuous remediation and reprovisioning loops.503 Service Unavailable or connection refused when attempting to reach aggregated APIs (such as metrics.k8s.io or system.antrea.io), and ImagePullBackOff errors occur for core components.This issue occurs when orphaned Role-Based Access Control (RBAC) resources (role and rolebinding for vmware-system-antrea-register in the default namespace) retain old kapp.k14s.io/app labels following a previous deployment or upgrade. The kapp-controller detects this label mismatch during reconciliation and aborts the Antrea package deployment to prevent overriding unowned resources. Consequently, the Antrea Container Network Interface (CNI) fails to initialize, preventing nodes from reaching a Ready state and triggering continuous Cluster API (CAPI) MachineHealthCheck remediation loops.
Remove the orphaned RBAC resources manually to allow the kapp-controller to recreate them with the correct application labels during the next reconciliation loop.
kubectl context to the affected workload cluster: kubectl config use-context <workload-cluster-context-name>kubectl delete role vmware-system-antrea-register -n defaultkubectl delete rolebinding vmware-system-antrea-register -n defaultkubectl patch pkgi <antrea-package-name> -n vmware-system-tkg --type merge -p '{"spec":{"paused":true}}'
kubectl patch pkgi <antrea-package-name> -n vmware-system-tkg --type merge -p '{"spec":{"paused":false}}'kubectl get pkgi -A
kubectl get nodes