After an upgrade of the management cluster to 2.2/2.3, may see the following error in the addons-controller pod when attempting to upgrade class based workload clusters:
kubecapture/core_v1/tkg-system/tanzu-addons-controller-manager-596998872d-cllkd/tanzu-addons-controller/tanzu-addons-controller.log:210:E0513 12:35:14.388006 1 clusterbootstrap_controller.go:749] ClusterBootstrapController "msg"="unable to create or patch PackageInstall default/<cluster-name>-kapp-controller for cluster: <cluster-name>-" "error"="kappcontrollerconfigs.run.tanzu.vmware.com \"<cluster-name>-kapp-controller-package\" not found
The cause for this issue is unknown. It is speculated that it is caused by a partial upgrade of the management cluster. When this issue occurs, critical configuration for some/all core components may be missing: CNI (AntreaConfig, ContourConfig), CSI (VSphereCSIConfig), CPI (VSphereCPIConfig) and KAPP (KappControllerConfig).
1). Check for the presence of AntreaConfig/ContourConfig (depending on the CNI that is currently in use), VSphereCSIConfig, VSphereCPIConfig and KappControllerConfig. Use a command similar to below to check if these objects exist:
kubectl get KappControllerConfig -n <cluster-namespace>
If any configuration is missing, contact Broadcom for assistance and reference this KB article.