Encrypted VM with Change Block Tracking (CBT) enabled unexpectedly powers off after shallow rekey operation
search cancel

Encrypted VM with Change Block Tracking (CBT) enabled unexpectedly powers off after shallow rekey operation

book

Article ID: 387897

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

When operating ESXi 7.x hosts connected to a vCenter 8.x and attempting to perform a shallow rekey of a powered-on, encrypted VM with CBT (Changed Block Tracking) enabled, the rekey operation will be successful, but the VM may unexpectedly power off (and subsequent HA actions conducted, if enabled). 

The following error may be seen in the ESXi /var/run/log/hostd.log:

2025-01-01T00:00:00.000Z verbose hostd[######] [Originator@#### sub=Vmsvc.vm:/vmfs/volumes/<datastore>/VM.vmx opID=CdrsLoadBalancer-######-########-##-##-##-####] Handling vmx message ####: Could not open or create change tracking file
--> Cannot open the disk '/vmfs/volumes/vmfs/volumes/datastore/VM.vmdk' or one of the snapshot disks it depends on.
--> An operation required the virtual machine to quiesce and the virtual machine was unable to continue running.

 

Environment

vCenter 8.x with ESXi 7.x hosts
vCenter 8.x with ESXi 8.x hosts

VM is encrypted
Change Block Tracking (CBT) is enabled

Note: this issue does not occur when using ESXi 7.x hosts connected to a vCenter 7.x.

Cause

The issue occurs after a successful rekey due to a logic in a race condition on ESXi between hostd and the VMX process when attempting to access the VM's vmdk disk(s) and the corresponding ctk file(s).

Resolution

This issue will be resolved in future versions of vSphere.


The following workarounds are available:

  • Power off the VM and perform the rekey operation
  • Disable CBT prior to rekeying the VM, rekey the VM and then reenable CBT.  (Note: an additional full VM backup should be taken after re-enabling CBT)
  • Connect the hosts from a vCenter 8.x back to a vCenter 7.x and perform the operation with the ESXi 7.x host(s) connected to a vCenter 7.x

Additional Information