Issue updating Tanzu guest cluster to v1.29 and above due to pre-installed snapshot-controller package

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

•  Tanzu guest cluster is being updated to v1.29 or above
•  There is an upstream "snapshot-controller" package already installed in the cluster
•  k get tkc shows 'Ready = False' from the Supervisor CLI
•  In the vSphere UI -> Supervisor -> Compute -> Tanzu Kubernetes Clusters, the TKC says "Updating"
• The Guest cluster shows the following message:

Message:               ClusterBootstrap conditions indicate it is still reconciling
OR
Message:               ClusterBootstrap conditions Vsphere-Pv-Csi-ReconcileFailed indicate reconcile has failed

• "kubectl describe clusterbootstrap" output shows the following:
      Last Transition Time:  yyyy-mm-ddThh:mm:ssZ
    Message:               kapp: Error: update deployment/snapshot-controller (apps/v1) namespace: kube-system: Updating resource deployment/snapshot-controller (apps/v1) namespace: kube-system:
  
  
• API Server under kube-system namespace : 
        Deployment.apps "snapshot-controller" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app":"snapshot-controller", "kapp.k14s.io/app":"###"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable (reason: Invalid)
    Status:      True
    Type:        Vsphere-Pv-Csi-ReconcileFailed

Environment

vSphere with Tanzu

Cause

This is caused by a conflicting installation of the snapshot-controller package. In TKr version 1.29 and above, the TKr comes with the snapshot-controller package pre-installed. If it is already deployed in the guest cluster, the update will get stuck.

Resolution

Remove the non-standard upstream installation of the snapshot-controller package from the Guest Cluster