Deleting a storage policy from the vCenter Server does not delete the associated Storage Class in the supervisor cluster

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

When deploying or managing services (such as CCI or Tanzu Kubernetes Grid) in a Supervisor Cluster, the following symptoms occur:

• Pod Status: Pods remain in a Pending state indefinitely.
  
  
• Pod Events: Running kubectl describe pod or kubectl get events reveals scheduling failures with the error below.
  
  vmodl.fault.SystemError or Failed to fetch content for the provided profiles.


• Log entries inside /var/log/vmware/vpxd/vpxd.log of the vCenter Server look like the following.
  
  SdrsPlacement: exception while filtering datastores by SPBM profile [UUID]: N5Vmomi5Fault11SystemError9ExceptionE(Fault cause: vmodl.fault.SystemError)
SOAP request returned HTTP failure; code: 500(Internal Server Error); fault: (vmodl.fault.InvalidArgument) { key = "com.vmware.pbm.pbmFault.locale", message = "Failed to fetch content for the provided profiles." }


• Log entries inside /var/log/vmware/vmware-sps/sps.log of the vCenter Server look like the following.
  
  com.vmware.pbm.util.PBMUtils - No providers found for some of the input profile ids: Profile not found. Id: [UUID]

Environment

VMware vSphere Kubernetes Service

Cause

This issue occurs when a vSphere Storage Policy is manually deleted from the vCenter Server UI while it is still associated with a StorageClass in the Supervisor Cluster.

Currently, there is no service within the Workload Control Plane (WCP) that monitors vCenter for storage policy deletions. Consequently, if a policy is removed from vCenter, the corresponding StorageClass and its reference in the Supervisor's internal metadata persist. When the system attempts to schedule pods using that storage class, the Storage Policy Based Management (SPBM) service fails to find the underlying profile ID in the vCenter Database (VCDB), leading to a system error.

Resolution

Deleting a storage policy from the vSphere UI that is currently in use by a Supervisor Cluster is unsupported and creates a stale configuration that is difficult to remediate. The below workaround might help if the policy is already deleted.

1. Identify the Stale StorageClass: Run the following command to find the StorageClass matching the deleted policy name.
   
   kubectl get sc
   
   
2. Delete the StorageClass: Manually delete the stale StorageClass from the Supervisor Cluster using the following command.
   
   kubectl delete storageclass <storage-class-name>


3. Restart Services: In some cases, internal caches in the WCP and SPS services must be cleared. Restart the following services on the vCenter Server Appliance.
   
   service-control --restart wcp
service-control --restart vmware-sps


4. Re-deploy Service: Attempt to re-deploy the affected service (e.g., CCI) ensuring it points to a valid, existing storage policy.

Note: If the issue persists after these steps, the stale profile ID may still be cached in the Supervisor's internal state. In such cases, contact Broadcom Support for further assistance.