Conflict when upgrading TKG guest cluster with Tanzu Application Platform installed with message "kube-system is already associated with a different label kapp.k14s.io"

Products

• VMware vSphere ESXi
• VMware vSphere with Tanzu

Issue/Introduction

All upgrades of TKG 1.0 to TKG 2.0 will fail if Tanzu Application Platform is installed. Furthermore, attempts to rectify the issue by uninstalling kapp-controller may result in the loss of production workloads.

Symptoms:
When upgrading a TKG 1.x cluster to TKG 2.x on vSphere 8 which has previously had either Tanzu Application Platform or Tanzu Cluster Essentials installed, the upgrade may stall with errors encountered when viewing the supervisor cluster.

kubectl describe tkc -n <namespace> <cluster_name> may show:

Message: 0/1 Control Plane Node(s) healthy. 3/3 Worker Node(s) healthy

A corresponding packageinstall resource <cluster_name>-kapp-controller will show the following errors:
 

status:
  conditions:
  - message: Error (see .status.usefulErrorMessage for details)
    status: "True"
    type: ReconcileFailed
  usefulErrorMessage: |-
    I1003 09:03:30.883788 23399 request.go:601] Waited for 1.002013393s due to client-side throttling, not priority and fairness, request: GET: https://192.x.x.x:6443/apis/apiextensions.k8s.io/v1
    kapp: Error: Ownership errors:
    - Resource 'rolebinding/pkgserver-auth-reader (rbac.authorization.k8s.io/v1) namespace: kube-system' is already associated with a different label 'kapp.k14s.io/app=...'

Environment

VMware vSphere 8.0 with Tanzu

Cause

Both Tanzu Application Platform and TKG 2.0 leverage installation of kapp-controller, with both attempting to lifecycle manage the installation.

Resolution

This will be fixed in a future release of VMware vSphere 8.

Workaround:
While logged into the guest cluster, run the following commands to remove the "kapp.k14s.io/app" label from objects managed by kapp-controller and delete namespace kapp-controller if it exists:
 

kubectl label rolebinding pkgserver-auth-reader kapp.k14s.io/app- -n kube-system
kubectl label customresourcedefinition internalpackagemetadatas.internal.packaging.carvel.dev kapp.k14s.io/app-
kubectl label clusterrolebinding kapp-controller-cluster-role-binding kapp.k14s.io/app-
kubectl label apiservice v1alpha1.data.packaging.carvel.dev kapp.k14s.io/app-
kubectl label clusterrole kapp-controller-user-role kapp.k14s.io/app-
kubectl label clusterrole kapp-controller-cluster-role kapp.k14s.io/app-
kubectl label clusterrolebinding pkg-apiserver:system:auth-delegator kapp.k14s.io/app-
kubectl label customresourcedefinition packagerepositories.packaging.carvel.dev kapp.k14s.io/app-
kubectl label customresourcedefinition packageinstalls.packaging.carvel.dev kapp.k14s.io/app-
kubectl label customresourcedefinition internalpackages.internal.packaging.carvel.dev kapp.k14s.io/app-
kubectl label customresourcedefinition apps.kappctrl.k14s.io kapp.k14s.io/app-
kubectl delete ns kapp-controller

 

Label removal can be verified by running kubectl get on the object with the --show-labels flag:

kubectl get rolebinding pkgserver-auth-reader --show-labels -n kube-system
kubectl get customresourcedefinition internalpackagemetadatas.internal.packaging.carvel.dev --show-labels
kubectl get clusterrolebinding kapp-controller-cluster-role-binding --show-labels
kubectl get apiservice v1alpha1.data.packaging.carvel.dev --show-labels
kubectl get clusterrole kapp-controller-user-role --show-labels
kubectl get clusterrole kapp-controller-cluster-role --show-labels
kubectl get clusterrolebinding pkg-apiserver:system:auth-delegator --show-labels
kubectl get customresourcedefinition packagerepositories.packaging.carvel.dev --show-labels
kubectl get customresourcedefinition packageinstalls.packaging.carvel.dev --show-labels
kubectl get customresourcedefinition internalpackages.internal.packaging.carvel.dev --show-labels
kubectl get customresourcedefinition apps.kappctrl.k14s.io --show-labels

Then continue upgrading as normal.

It is recommended to do this before upgrading guest clusters.