PVC Deletion Tasks Fail in vCenter Server with error: "The operation is not allowed in the current state."

Products

• VMware vCenter Server
• VMware vSphere Kubernetes Service
• VMware Cloud Native Runtimes

Issue/Introduction

In vCenter Server, vslm.vcenter.VStorageObjectManager.deleteVStorageObjectEx.label tasks are failing, and below entries related to the affected Persistent Volume Claims (PVCs) are observed in /var/log/vsan-health/vsanvcmgmtd.log

YYYY-MM-DDTHH:MM:SSZ info vsanvcmgmtd[11049] [vSAN@6876 sub=CnsVolMgr opId=0cad3899] CNS: UpdateVolumeMetadata with spec: (vim.cns.VolumeMetadataUpdateSpec) [
-->    (vim.cns.VolumeMetadataUpdateSpec) {
-->       volumeId = (vim.cns.VolumeId) {
-->          id = "#########"
-->       },
-->       metadata = (vim.cns.VolumeMetadata) {
-->          containerCluster = (vim.cns.ContainerCluster) {
-->             clusterType = "KUBERNETES",
-->             clusterId = "######",
-->             vSphereUser = "######",
-->             clusterFlavor = "VANILLA",
-->          },
-->          entityMetadata = (vim.cns.EntityMetadata) [
-->             (vim.cns.KubernetesEntityMetadata) {
-->                entityName = "pvc-#####-#####-#####-#####-######",
-->                clusterId = "######",
-->                entityType = "PERSISTENT_VOLUME",
-->             }
-->          ],
-->          containerClusterArray = (vim.cns.ContainerCluster) [
-->             (vim.cns.ContainerCluster) {
-->                clusterType = "KUBERNETES",
-->                clusterId = "#####",
-->                vSphereUser = "########",
-->                clusterFlavor = "VANILLA",
-->             }
-->          ]

Environment

VMware vCenter Server 7.x 

VMware vCenter Server 8.x

Cause

This typically occurs when the PVC is still being referenced by Kubernetes workloads or is in use by running pods, preventing the deletion operation from proceeding.

Resolution

To resolve this issue, follow these steps:

1. Check if the PVC is in use by any Kubernetes workloads:

• • Run the following command to check which pods are using the affected PVC:

kubectl get pods -n <namespace> -o json | jq '.items[] | select(.spec.volumes[].persistentVolumeClaim.claimName=="<pvclaimname>") | .metadata.name'

• • This will return the names of the pods using the PVC.
  • If the PVC is still in use, ensure that the pods or workloads referencing it are properly scaled down or deleted before attempting to delete the PVC again.

     2. Check for PVC references in Deployments, StatefulSets, or DaemonSets:

• • Run the following command to identify which workloads (Deployments, StatefulSets, or DaemonSets) are referencing the PVC:

kubectl get deployments,statefulsets,daemonsets -n <namespace> -o yaml | grep -B 5 -A 5 "claimName: <pvclaimname>"

• • This will show the deployments, statefulsets, or daemonsets that are using the PVC.

     3. Delete the PVC if it is no longer in use:

• • If you confirm that the PVC is not in use by any pods or workloads, you can proceed to delete the PVC:

kubectl delete pvc <pvcname> -n <namespace>

• • This will remove the PVC from the Kubernetes cluster, allowing the vStorage Object Manager task to succeed.

Additional Information

Ensure that all references to the PVC are removed in your Kubernetes environment to allow for successful deletion.