"<cluster>-kapp-controller-package\" not found" after upgrade from TKG 2.2 -> 2.3

Products

• Tanzu Kubernetes Grid

Issue/Introduction

After an upgrade of the management cluster to 2.2/2.3, may see the following error in the addons-controller pod when attempting to upgrade class based workload clusters: 

kubecapture/core_v1/tkg-system/tanzu-addons-controller-manager-596998872d-cllkd/tanzu-addons-controller/tanzu-addons-controller.log:210:E0513 12:35:14.388006       1 clusterbootstrap_controller.go:749] ClusterBootstrapController "msg"="unable to create or patch PackageInstall default/<cluster-name>-kapp-controller for cluster: <cluster-name>-" "error"="kappcontrollerconfigs.run.tanzu.vmware.com \"<cluster-name>-kapp-controller-package\" not found

 

 

Cause

The cause for this issue is unknown. It is speculated that it is caused by a partial upgrade of the management cluster. When this issue occurs, critical configuration for some/all core components may be missing: CNI (AntreaConfig, ContourConfig), CSI (VSphereCSIConfig), CPI (VSphereCPIConfig) and KAPP (KappControllerConfig). 

Resolution

1). Check for the presence of AntreaConfig/ContourConfig (depending on the CNI that is currently in use), VSphereCSIConfig, VSphereCPIConfig and KappControllerConfig. Use a command similar to below to check if these objects exist:

kubectl get KappControllerConfig -n <cluster-namespace> 

If any configuration is missing, contact Broadcom for assistance and reference this KB article.