"kapp: Error: update customresourcedefinition/volumegroupsnapshotclasses" Reconcile failed for a VKS Cluster's CSI package post VKR upgrade from 1.31 to 1.32 or higher

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Post VKS Cluster upgrade from 1.31 to 1.32, one package(pkgi) "<workload cluster name>-vsphere-pv-csi" is in "Reconcile failed" state
  
  
• While describing the failed pkgi, similar log entries will be seen
  • k describe pkgi -n vmware-system-tkg "<workload cluster name>-vsphere-pv-csi"
  • Useful Error Message:    kapp: Error: update customresourcedefinition/volumegroupsnapshotclasses.groupsnapshot.storage.k8s.io (apiextensions.k8s.io/v1) cluster:
      Updating resource customresourcedefinition/volumegroupsnapshotclasses.groupsnapshot.storage.k8s.io (apiextensions.k8s.io/v1) cluster:
        API server says:
          CustomResourceDefinition.apiextensions.k8s.io "volumegroupsnapshotclasses.groupsnapshot.storage.k8s.io" is invalid: status.storedVersions[0]:
            Invalid value: "v1alpha1": must appear in spec.versions (reason: Invalid)

Environment

vSphere Supervisor 9.x

vSphere Supervisor 8.x

VKS Service 3.2.0 and higher

Workload Cluster upgrading to KR v1.31 or higher

Cause

CRD volumegroupsnapshotclasses shipped with pv-csi v3.4.0 have version v1beta1.

CRD volumegroupsnapshotclasses.groupsnapshot.storage.k8s.io reported have version v1alpha1

Resolution

• Validate if the CRD is in use by running below command from Guest Cluster Context
  • kubectl get volumegroupsnapshotclasses --all-namespace
    
    
• If the command returns no output, it indicates that no resources are using this CRD. [Do not proceed to the next step if there are any resources in use, contact Broadcom Support Team]
  
  
• The CRD can be safely deleted using below commands
  • kubectl delete crd volumegroupsnapshotclasses.groupsnapshot.storage.k8s.io

    
• You can wait for 15 mins for kapp controller to reconcile the package (pkgi)
  
  
• Run the below command to verify if the pkgi is reconciled successfully.
  • kubectl get pkgi -A