Virtual machine backups fail due to locked files even if the VM is moved to other ESXi hosts in the cluster

Products

• VMware vSphere ESXi
• VMware vSphere ESX 8.x
• VMware vSphere ESX 7.x

Issue/Introduction

• Backup operations may work briefly after rebooting a VM with repeated failures, but they often fail again after a few scheduled or manual jobs.
  
  
• ESXi host logs at /var/run/log/hostd.log on the host where the VM is running may show consolidation failures
  

  YYYY-MM-DDTHH:MM:SS.287Z In(166) Hostd[<process_id>]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event <event_id>: Virtual machine <VM_Name> disks consolidation failed on <host_name> in cluster <cluster_name> in ha-datacenter.

• VM logs (vmware.log within the VM folder) may show:
  
  
  • Consolidation/file lock failure messages for specific VM disks (flat files or snapshot/delta files):
    

    YYYY-MM-DDTHH:MM:SS.266Z In(05) vcpu-0 <thread_id> DISKLIB-VMFS : "/vmfs/volumes/<UUID>/<VM_folder_name>/<VM_disk_name>-flat.vmdk" : closed. YYYY-MM-DDTHH:MM:SS.266Z In(05) vcpu-0 <thread_id> ConsolidateItemComplete: Online consolidation failed for disk node 'scsi0:2': Failed to lock the file (5). 

  • Issues opening or creating change tracking (ctk.vmdk) files, similar to:

    YYYY-MM-DDTHH:MM:SS.400Z In(05) vcpu-0 <thread-id> DISKLIB-CTK   : Could not open change tracking file "/vmfs/volumes/<UUID>/<VM_folder_name>/<VM_disk_name>.ctk.vmdk": Change tracking invalid or disk in use.
    YYYY-MM-DDTHH:MM:SS.402Z In(05) vcpu-0 <thread-id> DISKLIB-CTK   : Change tracking invalid; reinitializing.
    YYYY-MM-DDTHH:MM:SS.402Z In(05) vcpu-0 <thread-id> DISKLIB-CTK   : Auto blocksize for size 230686720 is 128.
    YYYY-MM-DDTHH:MM:SS.402Z In(05) vcpu-0 <thread-id> OBJLIB-FILEBE : Error creating file '/vmfs/volumes/<UUID>/<VM_folder_name>/<VM_disk_name>.ctk.vmdk'

 

Cause

Change tracking (ctk.vmdk) files are missing, corrupted, or mismatched with the VM's disks.

Resolution

• If CTK files are not present in the VM folder but log messages indicate failed attempts to open change tracking files, the issue can be resolved by disabling Changed Block Tracking (CBT) for the VM and then re‑enabling it. This forces regeneration of fresh CTK files. For details on CBT and how to enable or disable it, refer to Changed Block Tracking (CBT) on virtual machines
• If there are ctk.vmdk files present in the VM's files at /vmfs/volumes/<UUID>/<VM_folder_name>/ and VM logs indicate problems with opening invalid change tracking files, follow the resolution steps in Error: "the virtual disk is either corrupted or not a supported format" during snapshot operation to move CTK files to a temporary folder and run a snapshot consolidation.
  
  

Additional Information

• Operations such as Storage vMotion, VM cloning, or restoring VMs from a full backup are commonly used to return VM files to a clean base state to resolve various issues. These may not preserve or correctly regenerate a VM’s change tracking files (ctk.vmdk) in some cases, which are some of the ways to run into the issues described in this article.
• Troubleshooting issues when creating or committing snapshots in VMware ESXi/ESX is a good general resource for familiarization with symptoms associated with snapshot issues. As that is often what prevents successful backups, refer to the links for how to specifically resolve many frequent scenarios.
  
  
• Stale file locks on VMDKs left by 3rd party backup solutions