Unable to create PVC in VKS Cluster namespace

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Attempting to create a PVC and it gets stuck in a Pending state
• Doing a describe against the PVC from inside the Supervisor cluster (Troubleshooting vSphere Supervisor Control Plane VMs) has an Event of:
  

  Type = Warning
  Reason = ProvisioningFailed
  Message = failed to provision volume with StorageClass "<StorageClass_Name>": rpc error: code = InvalidArgument desc = StorageTopologyType is unset while topology label is present

Cause

VKS environment deployed in a pre-8.0 release then upgraded to 8.0 later causing the vSphere Zones to become available.  The storage policies associated with the VKS don't have the Storage Topology Type configured in the storage class (storage policy) that is used by the VKS environment.

Resolution

1. Access the Supervisor as per KB article 323407 (Troubleshooting vSphere Supervisor Control Plane VMs)
2. Save the StorageClass (SC) to a yaml.
3. Copy the yaml to a secondary file to create a backup.
4. Edit the yaml to remove the creationTimestamp, resourceVersion, and uid entries and add storageTopologyType: zonal under the Parameters section.
   
   It will look similar to:
   

   allowVolumeExpansion: true
   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     annotations:
       cns.vmware.com/StoragePoolTypeHint: cns.vmware.com/vsan
     name: <StorageClass_Name>
   parameters:
     storagePolicyID: <PolicyID>
     storageTopologyType: zonal
   provisioner: csi.vsphere.vmware.com
   reclaimPolicy: Delete
   volumeBindingMode: Immediate
   

5. Delete the existing storage class (kubectl delete sc <SC_Name>)
6. Recreate the SC using the yaml file you edited (kubectl apply -f edited_sc.yaml)
7. Go to the vSphere UI (Main Menu > Policies and Profiles > VM Storage Policies) and edit the policy you recreated.
   1. Select the “Enable Zonal Topology for Supervisor” on the Policy Structure page.
   2. If the Storage Compatibility page shows no compatible datastores go to the Incompatible Datastores tab and check the reason.  If the message is similar to: "Not support one or more required properties; datastore is not mounted on any available zone." then this indicates that vSphere Zones are not configured.
      1. If needed, cancel out and add a vSphere Zone (Main Menu > Inventory) navigate to the vCenter Configuration tab and select "vSphere Zones" and add a Zone.
   3. After confirming there are compatible datastores finish editing the policy and the default option "Reapply objects: Manually later" is sufficient when saving the policy.

 

Those steps should allow a PVC to be created.

If there are any questions or problems please reach out to Broadcom Support