Enabling and Persisting Changed Block Tracking (CBT) on Tanzu Kubernetes Grid Integrated Worker VMs for VADP-Based Backups
Article ID: 454008
Updated On:
Products
Issue/Introduction
Customers using VADP-based backup solutions (such as Dell PowerProtect Data Manager) with optimized/CBT-based backup for TKGI-provisioned Kubernetes clusters on vSphere may encounter Persistent Volume (PV) attach failures after a backup operation, TKGI upgrade, worker scale-out, or worker recreation. The symptom typically appears as pods stuck in ContainerCreating with an event such as:
FailedAttachVolume: AttachVolume.Attach failed for volume "pvc-xxxxx":
The operation is not allowed in the current state.
This occurs when Changed Block Tracking (CBT) is enabled on a First Class Disk (FCD)/Persistent Volume but not on the worker node VM (or vice versa) — a CBT state mismatch between the VM and its attached disk(s).
This article explains the supported method for enabling CBT on existing and future TKGI worker VMs using a BOSH vm_extension, why CBT can appear "enabled" yet still cause attach failures, and how to configure it so the setting persists across scaling, node repair, stemcell updates, and TKGI upgrades.
Environment
Tanzu Kubernetes Grid Integrated Edition
CBT
Cause
CBT on a vSphere VM is tracked at two independent levels:
VM-level flag — ctkEnabled, exposed to BOSH/TKGI via the vSphere CPI as a vmx_options parameter under a vm_extension.
Per-disk flag — scsiX:Y.ctkEnabled (the vSphere API equivalent is VirtualDisk.changeTrackingEnabled), tracked independently for each virtual disk attached to the VM.
Setting the VM-level flag via vm_extension does not retroactively enable the per-disk flag on disks that were already attached to the VM before the setting was applied. It only guarantees that new disks (e.g., PVs dynamically attached afterward via the vSphere CSI driver) inherit CBT tracking automatically at attach time.
This produces the mismatch:
A backup product (e.g., PowerProtect, Rubrik) enables CBT on the FCD/volume independently of the worker VM.
If the worker VM itself was not already CBT-enabled at the time the disk's tracking was turned on, or if the disk was attached before the VM-level setting existed, the VM and disk fall out of sync.
The vSphere/VADP layer treats this as an invalid state for subsequent disk operations, resulting in the "operation is not allowed in the current state" attach failure.
Resolution
Step 1: Enable CBT at the VM level via BOSH vm_extension:
Create a VM extension configuration file targeting the worker instance group (and any node pool instance groups in use), nesting ctkEnabled under vmx_options:
{
"instance_groups": [
{
"name": "worker",
"vm_extension": {
"vmx_options": {
"ctkEnabled": "TRUE",
"disk.enableUUID": "1"
}
}
}
]
}
Note: disk.enableUUID must always be included — omitting it will prevent volumes from attaching to the node at all, independent of CBT.
Apply the Configuration:
tkgi update-cluster CLUSTER-NAME --config-file config.json
This triggers a BOSH rolling update of the affected instance group. Because the setting is stored in the cluster's persisted instance-group configuration (not applied out-of-band), it is automatically re-applied by BOSH on every subsequent VM creation for that instance group — including scale-out, node repair/resurrection, stemcell updates, and TKGI control-plane upgrades. No manual re-application is required after these events.
Step 2: Verify VM-level CBT
In vCenter, open the worker VM's Edit Settings > VM Options > Advanced > Configuration Parameters and confirm:
| Key | Value |
| ctkEnabled | TRUE |
| disk.enableUUID | TRUE |
Step 3: Reconcile CBT on pre-existing disks
For PVs that were already attached to a worker before ctkEnabled was set at the VM level, the per-disk flag will not update automatically. Do one of the following:
- Preferred: Allow the backup product (e.g., PowerProtect) to run one backup cycle against the affected VM/disks. VADP-based backup products are expected to manage the per-disk changeTrackingEnabled flag themselves as part of initializing CBT-based backup, independent of the VM-level flag — this is standard behavior, not a workaround.
- Manual alternative:
Enable CBT directly on the FCD via the vCenter MOB:
- In a browser, navigate to:
https://<vc_fqdn>/mob/?moid=VStorageObjectManager&method=setVStorageObjectControlFlags- Populate the fields:
id: the Volume ID (FCD ID) of the affected disk
datastore: the ManagedObjectReference of the datastore holding the FCD
controlFlags: <controlFlags>enableChangedBlockTracking</controlFlags>
- Click Invoke Method.
- Verify the change by re-checking the FCD's changedBlockTrackingEnabled property, which should now report true.
Additional Information
Feedback
