VM CBT got auto disabled post powering on VM after a scheduled activity.

Products

• VMware vSphere ESXi 8.0

Issue/Introduction

You power off VM for any scheduled activiity.
Once you power on VM after the activity, VM`s CBT gets disabled and causing full backups.

In vmware.log we see multiple entries in different form with respect to cleanshutdown as false

yyyy-mm-ddThh:mm:ss In(05) vmx me2i1fzi-69334-auto-1hhz-h5:70015287-99-04-01-81-fbf6 DICT              scsi0:0.redo = ""
yyyy-mm-ddThh:mm:ss In(05) vmx me2i1fzi-69334-auto-1hhz-h5:70015287-99-04-01-81-fbf6 DICT                  vmci0.id = "464531068"
yyyy-mm-ddThh:mm:ss In(05) vmx me2i1fzi-69334-auto-1hhz-h5:70015287-99-04-01-81-fbf6 DICT             cleanShutdown = "FALSE"
yyyy-mm-ddThh:mm:ss In(05) vmx me2i1fzi-69334-auto-1hhz-h5:70015287-99-04-01-81-fbf6 DICT        scsi0:1.deviceType = "scsi-hardDisk"
yyyy-mm-ddThh:mm:ss In(05) vmx me2i1fzi-69334-auto-1hhz-h5:70015287-99-04-01-81-fbf6 DICT          scsi0:1.fileName = "BDDHKCCINB02_1-000001.vmdk"
yyyy-mm-ddThh:mm:ss In(05) vmx me2i1fzi-69334-auto-1hhz-h5:70015287-99-04-01-81-fbf6 DICT      sched.scsi0:1.shares = "normal"
yyyy-mm-ddThh:mm:ss In(05) vmx me2i1fzi-69334-auto-1hhz-h5:70015287-99-04-01-81-fbf6 DICT sched.scsi0:1.throughputCap = "off"
yyyy-mm-ddThh:mm:ss In(05) vmx me2i1fzi-69334-auto-1hhz-h5:70015287-99-04-01-81-fbf6 DICT           scsi0:1.present = "TRUE"
yyyy-mm-ddThh:mm:ss In(05) vmx me2i1fzi-69334-auto-1hhz-h5:70015287-99-04-01-81-fbf6 DICT              scsi0:1.redo = ""
yyyy-mm-ddThh:mm:ss In(05) vmx me2i1fzi-69334-auto-1hhz-h5:70015287-99-04-01-81-fbf6 DICT           config.readOnly = "FALSE"
yyyy-mm-ddThh:mm:ss In(05) vmx me2i1fzi-69334-auto-1hhz-h5:70015287-99-04-01-81-fbf6 DICT             uuid.location = "56 4d bc 62 f9 40 10 8c-9b 30 48 ca 74 0e 5c cb"
yyyy-mm-ddThh:mm:ss In(05) vmx me2i1fzi-69334-auto-1hhz-h5:70015287-99-04-01-81-fbf6 DICT                ctkEnabled = "TRUE"
yyyy-mm-ddThh:mm:ss In(05) vmx me2i1fzi-69334-auto-1hhz-h5:70015287-99-04-01-81-fbf6 DICT        scsi0:0.ctkEnabled = "TRUE"
yyyy-mm-ddThh:mm:ss In(05) vmx me2i1fzi-69334-auto-1hhz-h5:70015287-99-04-01-81-fbf6 DICT        scsi0:1.ctkEnabled = "TRUE"
yyyy-mm-ddThh:mm:ss In(05) vmx me2i1fzi-69334-auto-1hhz-h5:70015287-99-04-01-81-fbf6 DICT        scsi0:2.deviceType = "scsi-hardDisk"
yyyy-mm-ddThh:mm:ss In(05) vmx me2i1fzi-69334-auto-1hhz-h5:70015287-99-04-01-81-fbf6 DICT          scsi0:2.fileName = "BDDHKCCINB02_2-000001.vmdk"
yyyy-mm-ddThh:mm:ss In(05) vmx me2i1fzi-69334-auto-1hhz-h5:70015287-99-04-01-81-fbf6 DICT        scsi0:2.ctkEnabled = "TRUE"

Another such entry is seen under ConfigDB, this being sequential to the above entry

yyyy-mm-ddThh:mm:ss In(05) vmx - OvhdMem: memsize 24576 MB VMK fixed 3349 pages var(mem) 529 pages var(cpu) 22 cbrcOverhead 0 pages total 16409 pages
yyyy-mm-ddThh:mm:ss In(05) vmx - VMMEM: Precise Reservation: 629MB (MainMem=24576MB) VMK=64MB
yyyy-mm-ddThh:mm:ss In(05) vmx - Vix: [mainDispatch.c:1051]: VMAutomation_PowerOn. Powering on.
yyyy-mm-ddThh:mm:ss In(05) vmx - VMX_PowerOn: ModuleTable_PowerOn = 1
yyyy-mm-ddThh:mm:ss No(00) vmx - ConfigDB: Setting cleanShutdown = "FALSE"
yyyy-mm-ddThh:mm:ss No(00) vmx - ConfigDB: Setting softPowerOff = "FALSE"

Once we power on VM we see below entries related to CBT

yyyy-mm-ddThh:mm:ss No(00) vmx 1061c8f-01-d6-f6b1 ConfigDB: Unsetting "ctkEnabled"
yyyy-mm-ddThh:mm:ss No(00) vmx 1061c8f-01-d6-f6b1 ConfigDB: Unsetting "scsi0:0.ctkEnabled"
yyyy-mm-ddThh:mm:ss No(00) vmx 1061c8f-01-d6-f6b1 ConfigDB: Unsetting "scsi0:1.ctkEnabled"
yyyy-mm-ddThh:mm:ss No(00) vmx 1061c8f-01-d6-f6b1 ConfigDB: Unsetting "scsi0:2.ctkEnabled"
yyyy-mm-ddThh:mm:ss No(00) vmx 1061c8f-01-d6-f6b1 ConfigDB: Unsetting "scsi0:3.ctkEnabled"

Environment

ESXI 7.x
ESXI 8.x

Cause

The VM’s unclean shutdown  caused VMX to invalidate and wipe existing CTK metadata. Later, the backup/orchestration tool’s explicit CTK disable (API unsets at 13:04:19Z) prevented CTK from being reinitialized on new snapshots. As a result, subsequent incremental backups lacked CTK history and fell back to full-data transfers.

Resolution

Please make sure that the VM is gracefully shutdown
                                                           

1. Log in: 
   Access the vSphere Client (if using vCenter) or the ESXi host's web interface directly. 
2. Select the VM: 
   In the inventory pane, locate and select the virtual machine you wish to shut down. 
3. Shut Down:

         Graceful Shutdown: Right-click the virtual machine and navigate to Power > Shut Down Guest OS. This initiates a clean shutdown from within the guest operating system.