"Error: The operation is not allowed in the current state. govc: The operation is not allowed in the current state." when deleting FCD Snapshots via govc

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• When attempting to delete First Class Disk (FCD) snapshots using the `govc` client, the operation fails with the following error:
  
  govc: The operation is not allowed in the current state.
  
  
• The /var/log/vmware/vpxd/vpxd.log of the vCenter Server Appliance looks like the following. 
  
      error vpxd[53067] [VpxLRO] -- ERROR task-12367968 -- VStorageObjectManager -- vim.vslm.vcenter.VStorageObjectManager.deleteSnapshot: :vim.fault.InvalidState
    Result: (vim.fault.InvalidState) { msg = "The operation is not allowed in the current state." }


• Per /var/run/log/hostd.log of the ESXi Host, the delete operation fails due to an invalid transition of the Virtual Machine state. The log entry looks like the following. 
  
  Invalid transition requested (VM_STATE_BRANCH_DISK -> VM_STATE_CONSOLIDATE_DISK_CHAIN): Invalid state


• This indicates that the VM is already undergoing a snapshot operation (such as `createVStorageObjectSnapshot`) on another disk, which holds the VM state lock.

Environment

VMware vCenter Server
VMware vSphere ESXi
VMware vSphere Kubernetes Service
VMware vSphere CSI driver

Cause

This issue occurs due to concurrency conflicts where a third-party backup tool (e.g., Cohesity using the Velero plugin) is aggressively triggering `VolumeSnapshot` requests. In a VKS environment, the call chain is as follows:

1. Third-party tool creates `VolumeSnapshot` objects in the Guest Cluster.
2. Guest CSI Controller triggers a snapshot request to the Supervisor Cluster.
3. Supervisor CSI Controller passes the request to CNS (Cloud Native Storage), which invokes the VSLM API to create the FCD snapshot.
4. If the backup tool continuously creates new snapshot objects (often due to retries from previous failures like `TOOMANYREDO`), the VM state remains locked, preventing manual cleanup via `govc`.

Resolution

To resolve this, you must stop the incoming snapshot requests to release the VM state lock before performing a manual cleanup.

Manual Cleanup Method:

Step 1: Identify and Stop Backup Workflows

1. Check for existing `VolumeSnapshot` objects in the affected Guest Cluster namespace using the command below.
   
   kubectl get volumesnapshot -n <namespace>


2. Delete any remaining `VolumeSnapshot` objects.
   
   kubectl delete volumesnapshot --all -n <namespace>


3. Pause or disable the third-party backup schedules to prevent new objects from being created during cleanup.

Step 2: Scale Down CSI Controllers.

To ensure no automated system triggers a snapshot during manual cleanup, scale down the CSI controllers in both the Guest and Supervisor clusters using the command below.

kubectl scale deployment vsphere-csi-controller -n vmware-system-csi --replicas=0

Step 3: Perform Manual Cleanup.

With the controllers scaled down, retry the `govc` snapshot removal command:

govc disk.snapshot.rm -dc <datacenter> -ds <datastore> <fcd-id> <snapshot-id>

Step 4: Restore Services.

Once the stale snapshots are removed, scale the controllers back up on both the Supervisor and Guest Cluster.

kubectl scale deployment vsphere-csi-controller -n vmware-system-csi --replicas=1 (for Guest cluster) 
kubectl scale deployment vsphere-csi-controller -n vmware-system-csi --replicas=3 (for Supervisor cluster)