Virtual Machines with CBT enabled running on vVOLs may go into Hang or a Guest OS crash post VMotion failure - Cannot enable changed block tracking and multi-writer sharing at the same time.

Products

• VMware vSphere ESXi 8.0
• VMware vSphere ESXi

Issue/Introduction

Symptoms 

• Post VMotion migration the Virtual Machine either hangs / Crashes where in vSphere HA will restart the guest OS 
  
  
• The vCenter UI will report the message - "Cannot enable changed block tracking and multi-writer sharing at the same time."
  
  
• The vCenter UI will also show the message as "Migration failed after VM memory precopy. failed to reopen swapfile on destination. Verify that the swapfile storage location is accessible from both source and destinations hosts. Error status: Failure. Failed to resume destination VM"
  
  
• Hostd Log shows the below error message 
  Note - The Virtual machine disks residing on a vVOL datastores are downgraded to multi-writer mode during vMotion. 
  
  YYYY-MM-DD|HH:MM:SS.761Z Db(167) Hostd[2100706] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vvol:XXXXXXXXX-XXXXXXXXXX/naa.123/VM-Name.vmx] Ignored toolsManifestInfo update of size 0
  YYYY-MM-DD|HH:MM:SS 206Z In(166) Hostd[2100728] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vvol:XXXXXXXXX-XXXXXXXXXX/naa.123/VM-Name.vmx] VigorMigrateNotifyCb:: hostlog state changed from emigrating to failure
  YYYY-MM-DD|HH:MM:SS..206Z Db(167) Hostd[2100728] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vvol:XXXXXXXXX-XXXXXXXXXX/naa.123/VM-Name.vmx] Handling vmx message 47043560: Cannot enable changed block tracking and multi-writer sharing at the same time.
  YYYY-MM-DD|HH:MM:SS..206Z Db(167) Hostd[2100684] --> Cannot open the disk '/vmfs/volumes/vvol:XXXXXXXXX-XXXXXXXXXX/naa.123/VM-Name-000003.vmdk' or one of the snapshot disks it depends on.
  YYYY-MM-DD|HH:MM:SS..207Z Wa(164) Hostd[2100728] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vvol:XXXXXXXXX-XXXXXXXXXX/naa.123/VM-Name.vmx] Failed to find activation record, event user unknown.
  YYYY-MM-DD|HH:MM:SS..207Z In(166) Hostd[2100728] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 44470 : Error message on [VM-Name.vmx] on [ESX Host name] in ha-datacenter: An operation required the virtual machine to quiesce and the virtual machine was unable to continue running.
  
  

Conditions 

• The Virtual Machine does not have "multi-writer" or Disk sharing enabled within the VMX settings
  One can verify this with the steps mentioned in this KB article - 313527
  
  
• The Virtual Machines running on vVOL datastores enabled with CBT
  One can verify CBT enabled option using this KB article - 320557
  
  
• The issue occurs only when vMotion fails on destination ESXi host post stun and Virtual Machine has to be resumed on source host.

Environment

vSphere 8.x

 

Cause

• The issue occurs when the "Migrate_Restoring() is false for source vmx, as a result DiskIsLockDowngradedByVMotionDest is false, this ends failing the VMotion and crashing the guest OS virtual machine.

Note - The Virtual machine restarted by the vSphere HA restart is just a post incident recovery mechanism as the VMX process in the case will be unresponsive. "vmx does not appear to be running" This will be the message seen in the Hostd Logs.  

Resolution

Current Status 

• Broadcom Engineering is aware of the issue and is working on creating a fix.

Workaround 

• Disable the Change Block tracking for the guest OS to address the issue
  
  
• Steps to disable CBT, refer the KB article - 320557 section "To disable CBT"