"The operation is not allowed in the current state" error when trying to attach a Persistent Volume to a Kubernetes worker node

Products

• VMware vCenter Server
• VMware vSphere Kubernetes Service

Issue/Introduction

Each of the following conditions should be met for the issue to match this particular article, if some of them are not met then the cause may be different and should be investigated further.

• First Class Disks (FCDs) provisioned via the vSphere Container Storage Plug-in (vSphere CSI driver) fail to attach to Virtual Machines.
• Dynamic storage environments (seen in Kubernetes/OpenShift): CBT mismatch issues may occur in environments with automated storage provisioning where FCDs are dynamically attached and detached from VMs. Container platforms like OpenShift may display CBT mismatch alerts that prevent new storage provisioning through the Container Storage Interface (CSI) driver.
• In environments with automated storage provisioning (such as Kubernetes/OpenShift, vRealize Automation, or other cloud management platforms), you may also observe:
  
  
  • Platform-specific alerts: CBT mismatch warnings in orchestration platform consoles (e.g., OpenShift Console, vRealize, etc.)
  • Automated storage provisioning failures: Dynamic volume attachment operations fail due to CBT configuration mismatches
  • Application deployment issues: Workloads requiring persistent storage fail to start or remain in pending state due to storage attachment failures
  
  Additional log locations for dynamic storage environments:
  
  • Container Storage Interface (CSI) driver logs showing CBT-related attachment failures
  • Cloud management platform logs showing persistent volume attachment errors
  • Orchestration platform events showing storage provisioning failures

In Kubernetes environment:

• Running a describe on the impacted Persistent Volume in Kubernetes may show the following event:

CnsFault error: CNS: Failed to attach disk when calling AttachDisk:Fault cause: vim.fault.InvalidState\\n\"\n})\n"

 

In the vSphere environment the following can be observed:

• The same failure also occurs if the user attempts to manually attach the disk to the Virtual Machine via the vCenter Server WebClient

• In vCenter Server Checking the Cloud Native Storage (CNS) logs at /var/log/vmware/vsan-health/vsanvcmgmtd.log at the time of the failure show the following error:

error vsanvcmgmtd[#####] [vSAN@#### sub=FcdService opId=########] CNS: Failed to attach disk when calling AttachDisk:N3Vim5Fault12InvalidState9ExceptionE(Fault cause: vim.fault.InvalidState
error vsanvcmgmtd[#####] [vSAN@#### sub=VolumeManager opId=########] CNS: Failed to attach disk to vm: vim.VirtualMachine:vm-#### with err: N3cns12CnsExceptionE(CNS: Failed to attach disk when calling AttachDisk:Fault cause: vim.fault.InvalidState

• In vCenter Server 'vim.fault.InvalidState' error is also reported in the /var/log/vmware/vpxd/vpxd.log as follows:

info vpxd[#####] [Originator@#### sub=Default opID=#########-##] [VpxLRO] -- ERROR task-####### -- vm-###### -- vim.VirtualMachine.attachDisk: vim.fault.InvalidState:
--> Result:
--> (vim.fault.InvalidState) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>
-->    msg = "The operation is not allowed in the current state."
--> }
--> Args:
-->
--> Arg diskId:
--> (vim.vslm.ID) {
-->    id = "######-####-####-########"
--> }
--> Arg datastore:
--> 'vim.Datastore:datastore-<MOID>'
--> Arg controllerKey:
--> 1000
--> Arg unitNumber:
-->

 

• From ESXi hostd logs where the worker node resides you would also see error:

verbose hostd[########] [Originator@#### sub=Vmsvc.vm:/vmfs/volumes/<datastore>/<path>/<node>.vmx  opID=########-####-####-####-############-######-##-##-#### user=vpxuser:VirtualCenter] Cannot attach a CBT enabled fcd to CBT disabled VM

 

info hostd[########] [Originator@#### sub=Vmsvc.vm:/vmfs/volumes/<datastore>/<path>/<node>.vmx opID=########-####-####-####-############-######-##-##-#### user=vpxuser:VirtualCenter] Reconfigure failed: N3Vim5Fault12InvalidState9ExceptionE(Fault cause: vim.fault.InvalidState]

 

The following can confirm Changed Block Tracking settings on the VM and Storage Object:

• Changed Block Tracking (CBT) is not enabled on the Kubernetes worker node Virtual Machine where the disk is attempting to be attached.
  • This can be checked by right-click the Virtual Machine and navigate to Edit Settings > VM Options > Advanced > Configuration Parameters > Edit Configuration and filter for ctkEnabled.
  • NOTE: If this parameter is either missing or set to "FALSE" then CBT is not enabled on the Virtual Machine.
• Changed Block Tracking is enabled on the First Class Disk which is failing to attach.
  • This can be checked via the vCenter Server Managed Object Browser (mob) by opening the following URL in a web browser:
    https://<vc_fqdn>/mob/?moid=VStorageObjectManager&method=retrieveVStorageObject.
  • In the window that opens you need to enter the volume ID and datastore managed object reference.
    • The volume ID can be retrieved by selecting the Datastore or Datastore cluster in the WebClient inventory and select:
      • Monitor > Cloud Native Storage > Container Volumes
      • Click the Details icon on the impacted Volume Name > Basics > Note the Volume ID. 
      • While the datastore managed object reference can be retrieved either from the vpxd logged message just like the message as shown in the above vpxd example in green or by selecting the datastore where the volume resides in the WebClient inventory and copying the "datastore-MOID" reference from the browser URL (e.g. https://vc-fqdn/ui/app/datastore;nav=s/urn:vmomi:Datastore:datastore-MOID:########-####-####-####-############/summary)
  • Once the id & datastore details are populated in the pop-up window then click Invoke Method and confirm the 'changedBlockTrackingEnabled' value which is returned shows as true

Environment

VMware vCenter Server: 7, 8
vSphere Supervisor
Cloud Native Storage
TKGm: 2.4.1

 

Cause

 

• Change Block Tracking (CBT) is enabled on the First Class Disk (FCD) but not on the Kubernetes node VM
• CBT becomes enabled on the disks if they are attached to Virtual Machines being backed up that have CBT enabled. Detaching the FCD from the Virtual Machine does not disable CBT on the disk.
• Attaching an FCD with CBT enabled to a VM with CBT disabled throws an error, unless the FCD is attached as "independent nonpersistent" disk.

Common scenario in dynamic storage environments with backup solutions:

• CBT becomes enabled on First Class Disks when they are attached to VMs that have CBT enabled, such as backup proxy VMs or backup appliances.
• Even when target VMs (e.g., Kubernetes worker nodes, application VMs) are excluded from backup policies, the FCDs themselves retain the CBT setting from their previous attachment to CBT-enabled infrastructure.
• When automated storage provisioning systems subsequently attempt to attach these FCDs to VMs that don't have CBT enabled, the mismatch condition occurs. This scenario is particularly common in:
  • Container environments (Kubernetes/OpenShift) where worker VMs typically don't have CBT enabled
  • Cloud automation platforms that dynamically provision storage
  • Any environment where FCDs are shared or moved between different VM types with varying CBT configurations

Resolution

Apply one of the following options:

Disable Changed Block Tracking on the First Class Disk. To do this perform the following steps

1. Go to https://<vc_fqdn>/mob/?moid=VStorageObjectManager&method=clearVStorageObjectControlFlags
2. Provide the Volume id in the id field along with the datastore ManagedObjectReference in the datastore field
3. Provide enableChangedBlockTracking in the controlFlags field as follows
   <controlFlags>enableChangedBlockTracking</controlFlags>
4. Then click on Invoke Method
5. Recheck the FCD changedBlockTrackingEnabled setting via MOB again to ensure it now shows as false as outlined in the Symptoms section above.

 

OR

• If CBT is being enabled on the FCD due to a backup solution, and if policy/process prevents you from the previous 2 options, reconfigure the backup solution to not use CBT (if available).
  
  

Additional Information

Impact/Risks:
Please consult with your backup vendor for any possible impacts if you decide to disable Changed Block Tracking on the Virtual Machines as part of the resolution.

Error should no longer seen from vSphere 8.0U1 and above, after this, the CBT state of VM will be enabled and the AttachDisk will be retried
From 9.0 and above CBT state of VM will be independent of the CBT state of FCD and the AttachDisk will proceed as usual (attach disk will not fail with CBT mismatch error)