CBT Reset Observed on VM Due to Backup Application API Calls

Products

VMware vSphere ESXi

Issue/Introduction

VMware virtual machines experience unexpected resets to Changed Block Tracking (CBT).
Incremental backups fail or automatically convert to full backups as CBT is disabled and Recovery Point Objective (RPO) is impacted due to increased backup duration.
vCenter events show config.changeTrackingEnabled: true -> false and in journalctl logs in vCenter, you see events similar to:

YYYY-MM-DDTHH:MM:SS <vCenter_FQDN> vpxd[33157]: Event [205598701] [1-1] [YYYY-MM-DDTHH:MM:SS.] [vim.event.VmReconfiguredEvent] [info] [<domain/user_name>] [Datacenter Name] [205598700] [Reconfigured <VM_Name> on <ESXi_Host_FQDN> in Datacenter Name.
Modified:
config.changeTrackingEnabled: true -> false;
Added:
Deleted:
config.extraConfig("ctkEnabled"): (key = "ctkEnabled", value = "TRUE");
config.extraConfig("scsi0:0.ctkEnabled"): (key = "scsi0:0.ctkEnabled", value = "TRUE");
]
.
.
.
YYYY-MM-DDTHH:MM:SS <vCenter_FQDN> vpxd[33157]: Event [205628830] [1-1] [YYYY-MM-DDTHH:MM:SS.] [vim.event.VmReconfiguredEvent] [info] [<domain/user_name>] [Datacenter Name] [205628822] [Reconfigured <VM_Name> on <ESXi_Host_FQDN> in Datacenter Name.
Modified:
config.changeTrackingEnabled: false -> true;
Added:
config.extraConfig("ctkEnabled"): (key = "ctkEnabled", value = "TRUE");
config.extraConfig("scsi0:0.ctkEnabled"): (key = "scsi0:0.ctkEnabled", value = "TRUE");
Deleted:
]
In /var/log/vmware/vpxd/vpxd.log on vCenter, you see entries similar to:

YYYY-MM-DDTHH:MM:SS. info vpxd[35614] [Originator@6876 sub=vpxLro opID=31bd3###] [VpxLRO] -- BEGIN task-8830483 -- vm-<vm_id>-- vim.VirtualMachine.reconfigure -- 5292b268-e3e6-ff9e-bda8-############(52c17ea6-e958-c247-18b1-############)
YYYY-MM-DDTHH:MM:SS. info vpxd[35614] [Originator@6876 sub=vpxLro opID=31bd3###-01] [VpxLRO] -- BEGIN lro-2570878 -- vm-<vm_id>-- vim.VirtualMachine.reconfigure -- 5292b268-e3e6-ff9e-bda8-############(52c17ea6-e958-c247-18b1-############)
YYYY-MM-DDTHH:MM:SS. info vpxd[35614] [Originator@6876 sub=VmCheck opID=31bd3###-01] CompatCheck results: (vim.vm.check.Result) [
--> (vim.vm.check.Result) {
--> vm = 'vim.VirtualMachine:b8aa5b8a-3771-4720-aa20-############:vm-<vm_id>',
--> host = 'vim.HostSystem:b8aa5b8a-3771-4720-aa20-############:host-id',
--> }
--> ]
YYYY-MM-DDTHH:MM:SS. info vpxd[35614] [Originator@6876 sub=VmProv opID=31bd3###-01] Applying ConfigSpec (vim.vm.ConfigSpec) {
--> createDate = "YYYY-MM-DDTHH:MM:SS.",
--> changeTrackingEnabled = false

Profiler output confirms the client IP making the API request (/var/log/vmware/vpxd/vpxd-profiler.log):

/SessionStats/SessionPool/Session/Id='5292b268-e3e6-ff9e-bda8-############'/Username='<user_name>'/ClientIP='<source_ip_address>'/PropertyCollector/FilterCount/total 0
The vmware.log(/vmfs/volumes/Datastore-UUID/VMFolder) of the impacted VM contains entries showing configuration changes initiated by an external session (e.g., __GX_BACKUP__ for Commvault).

YYYY-MM-DDTHH:MM:SS No(00) vmx - ConfigDB: Unsetting "ctkEnabled"
YYYY-MM-DDTHH:MM:SS In(05) vmx - VigorTransport_ServerSendResponse opID=2319441f-01-##-#### seq=309353: Completed MiscOpts.SetCtkEnabled request in 3564 US.
YYYY-MM-DDTHH:MM:SS In(05) vmx - VigorTransportProcessClientPayload: opID=2319441f-01-##-#### seq=309354: Receiving NVDimm.SetGlobalCtkEnabled request.
YYYY-MM-DDTHH:MM:SS In(05) vmx - VigorTransport_ServerSendResponse opID=2319441f-01-##-#### seq=309354: Completed NVDimmVigor.SetGlobalCtkEnabled request in 19 US.
YYYY-MM-DDTHH:MM:SS In(05) vmx - VigorTransportProcessClientPayload: opID=2319441f-01-##-#### seq=309356: Receiving Disk.SetCtkEnabled request.
YYYY-MM-DDTHH:MM:SS No(00) vmx - ConfigDB: Unsetting "scsi0:0.ctkEnabled"
YYYY-MM-DDTHH:MM:SS In(05) vmx - VigorTransport_ServerSendResponse opID=2319441f-01-##-#### seq=309356: Completed DiskEarlyDevice.SetCtkEnabled request in 3788 US.
YYYY-MM-DDTHH:MM:SS In(05) vmx - VigorTransportProcessClientPayload: opID=2319441f-01-##-#### seq=309362: Receiving HotPlugManager.EndBatch request.
YYYY-MM-DDTHH:MM:SS In(05) vmx - VigorTransport_ServerSendResponse opID=2319441f-01-##-#### seq=309362: Completed HotPlugManager.EndBatch request in 12 US.
YYYY-MM-DDTHH:MM:SS In(05) vmx - VigorTransportProcessClientPayload: opID=61b91ba7-##-#### seq=309489: Receiving Snapshot.Take request.
YYYY-MM-DDTHH:MM:SS In(05) vmx - SnapshotVMX_TakeSnapshot start: '__GX_BACKUP__', deviceState=0, lazy=0, quiesced=0, forceNative=0, tryNative=1, saveAllocMaps=0

Environment

vCenter Server 7.x
vCenter Server 8.x

Cause

CBT resets are not initiated by VMware components. Resets occur when 3rd-party backup applications send authenticated API calls to vCenter to unset the ctkEnabled parameter. This often happens if the backup application detects inconsistent metadata or snapshot chain issues.

Resolution

Identify the Initiator: Review vCenter events or vpxd.log to identify the service account or Client IP address initiating the VmReconfiguredEvent. Log entries will show: changeTrackingEnabled = false.
Engage Backup Vendor Support: Request an investigation into why the application is triggering CBT resets. Provide the vmware.log and vpxd.log snippets showing the API-driven reconfiguration.
Re-baseline CBT:
- Disable and re-enable CBT on the impacted VM. Refer to Enabling or disabling Change Block Tracking (CBT) .
- Run a manual full backup to reset the tracking ID.
Monitor for occurrence: Track vCenter events for further MiscOpts.SetCtkEnabled requests to ensure the 3rd-party application is no longer unsetting the parameter.

Additional Information

For further details on CBT, refer to Changed Block Tracking (CBT) on virtual machines .
To speak with a customer representative or a Support Engineer see . Scroll to the bottom of the page and click on your respective region.