TKGm Storage Policy not always is applied to default storage class on workload clusters

Products

VMware Tanzu Kubernetes Grid Management

Issue/Introduction

When creating a workload cluster it was discovered that the default storage class created during cluster creation does not contain the expected storage policy.

kubectl get sc default -oyaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kapp.k14s.io/identity: v1;/storage.k8s.io/StorageClass/default;storage.k8s.io/v1
...
    kapp.k14s.io/update-strategy: skip
    storageclass.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2024-08-20T07:56:59Z"
  labels:
.......
  name: default
provisioner: csi.vsphere.vmware.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

As seen from the command the expected parameter is missing

  parameters:
    storagepolicyname: test-vsan-pol

Environment

TKGm 2.5.1

Cause

It was discovered that the switch VSPHERE_STORAGE_POLICY_ID is not defined in the clusterbootstrap object during cluster creation.

Below is described in more details the correlation between the switch and the process how this is applied.

When cluster is created using cluster class there are several components that have to be defined:

kind: VSphereCPIConfig
kind: VSphereCSIConfig
kind: ClusterBootstrap
kind: Secret
kind: Cluster

In case the cluster object is created without cluster bootstrap definition created this object will be automartically created with default setting without taking into account the switch VSPHERE_STORAGE_POLICY_ID

Once the cluster is created the KAPP controller will deploy default storage class as per the secret defined in the MGMT cluster

The reconciliation process follows the below hierarchy

clusterboostrap when initially defined must contain the switch in place for the cluster API be able to successfully reconcile the default storage class
- secret defined in the MGMT cluster in usually default namespace with name: <worklaod-clustername>-tkg-storageclass-package is generated
- Secret <worklaod-clustername>-tkg-storageclass-data-valuesname updates from above automatically via KAPP once the workload cluster is created
  - Packageinstall deployed on workload cluster in namespace tkg-system <worklaod-clustername>-tkg-storageclass reads the above secret and run reconciliation
    - APP managed by the above package installed in namespace tkg-system <worklaod-clustername>-tkg-storageclass
    - The App deploys the default storage class based on the parameters defined in the clusterbootstrap
    - If the storage class already exists the object will not be reconciled

Clusterbootstrap extract:

  - refName: tkg-storageclass.tanzu.vmware.com.0.32.2+vmware.1
    valuesFrom:
      inline:
        infraProvider: vsphere

Clusterboostrap with the switch defined:

  - refName: tkg-storageclass.tanzu.vmware.com.0.32.2+vmware.1
    valuesFrom:
      inline:
        VSPHERE_STORAGE_POLICY_ID: test-vsan-pol
        infraProvider: vsphere

Resolution

Based on the https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/2.5/using-tkg/workload-clusters-storage.html#default-storage-class-2

Do not modify the default storage class definition. To customize a storage class, create a new StorageClass definition with a different name instead of modifying the default object created by TKG.

Recommended approach would be to either recreate the cluster with correct switch or create a new storage class with the desired parameters in place and complete the provisioning from the new storage class.

Additional Information

https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/2.5/using-tkg/workload-clusters-storage.html#default-storage-class-2