vSphere TLS thumbprint update using tanzu credentials plugin doesn't update all the cluster objects in Tanzu Kubernetes Grid Management 2.3.x and above

Products

• Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid 1.x
• VMware Tanzu Kubernetes Grid Management
• VMware Tanzu Kubernetes Grid Plus
• VMware Tanzu Kubernetes Grid Plus 1.x

Issue/Introduction

After updating the vSphere TLS thumbprint using tanzu credentials plugin as mentioned in the doc Update Cluster Credentials, you would notice vsphere-csi-controller pods in crashloop state

Environment

TKGm 2.3.x
TKGm 2.4.x
TKGm 2.5.x

Cause

Updating the thumbprint doesn't update vspherecsiconfig CR for the classy cluster due to which the csi pods doesn't come up

Resolution

1. Check the vsphere thumbprint that's present before the update
   kubectl get vspherecluster -A  -o yaml | grep thumbprint
   
   
2. After rotating the vCenter certificates, you will notice the vsphere-csi-controller pods crashing.
   kubectl  -n vmware-system-csi logs -l app=vsphere-csi-controller -c vsphere-csi-controller


3. Also csi pkgi will be in the Reconcile Failed state
   kubectl -n tkg-system get apps mgmt-cluster-vsphere-csiYou can see the same status for workload clusters.
   
   
4. Updating the Credentials using the Tanzu CLI
   tanzu mc credentials update --vsphere-thumbprint 55:94:20:<Snip>:1A:51:9A:0F:98 --cascading -v 9


5. In the legacy workload cluster, delete the vsphere-csi pkgi, which will be recreated and nodes reregistered; csinode and csinodetopologies objects will be created with the correct thumbprint.
   kubectl get pkgi -A
kubectl delete pkgi vsphere-csi -n tkg-system
kubectl get pkgi -A
kubectl -n vmware-system-csi get po
   
   
6. Classy cluster, needed to update the vspherecsiconfig CR with the new thumbprint, by setting the context to the Management cluster and running the commands below:
   kubectl edit vspherecsiconfigs.csi.tanzu.vmware.com -n tkg-system mgmt-cluster
kubectl edit vspherecsiconfigs.csi.tanzu.vmware.com testcluster


7. Once the vsphere-csi config object is updated, proceed with deleting the vsphere-csi pkgi for the respective clusters
   
   
8. On the Classy workload cluster,
   kubectl get pkgi -A
kubectl delete pkgi testcluster-vsphere-csi -n tkg-system packageinstall.packaging.carvel.dev "testcluster-vsphere-csi" deleted
kubectl get pkgi -A
   
   
9. On the Management cluster 
   kubectl get pkgi mgmt-cluster-vsphere-csi -n tkg-system
kubectl delete pkgi mgmt-cluster-vsphere-csi -n tkg-system packageinstall.packaging.carvel.dev "mgmt-cluster-vsphere-csi" deleted
kubectl get pkgi mgmt-cluster-vsphere-csi -n tkg-system
kubectl -n vmware-system-csi get po