When attempting to take a snapshot of a virtual machine on a VVOL datastore, the virtual machine crashes due to a CID mismatch.

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• The virtual machine is running on a VMware vSphere Virtual Volumes (VVols) datastore.
• When you attempt to take, delete, or consolidate a virtual machine snapshot, the virtual machine may be unexpectedly shut down. The following entries appear in the vmware.log:
  

  YYYY-MM-DDTHH:mm:SS.SSSZ In(05) vcpu-0 - DISK:bora/devices/disk/disk.c:7490 contentID mismatch on 'scsi0:0' - was 0x1234ABCD, is 0x2345BCDE 
  YYYY-MM-DDTHH:mm:SS.SSSZ In(05) vcpu-0 - [msg.checkpoint.continuesync.error] An operation required the virtual machine to quiesce and the virtual machine was unable to continue running. 

  CID (contentID) is a unique identifier for managing the snapshot chain. In this example, the expected CID is 0x1234ABCD, but the actual CID is 0x2345BCDE.
  Note: This issue can also occur when taking the first snapshot — that is, when no snapshots are in use.

• The virtual machine was vMotioned before the snapshot operation.
•  During the vMotion process, the guest OS performed a "First Write" to a disk that has encountered a CID mismatch. The following entries appear in the vmware.log: 
  

  YYYY-MM-DDTHH:mm:SS.SSSZ In(05) vmx - MigrateSetState: Transitioning from state MIGRATE_TO_VMX_PREPARING (2) to MIGRATE_TO_VMX_PRECOPY (3).
  YYYY-MM-DDTHH:mm:SS.SSSZ In(05) vcpu-0 - HBACommon: First write on scsi0:0.fileName='/vmfs/volumes/vvol:################-0123456789abcdef/naa.60##########################0001/vm_name.vmdk'
  YYYY-MM-DDTHH:mm:SS.SSSZ In(05) vcpu-0 - MigrateSetState: Transitioning from state MIGRATE_TO_VMX_PRECOPY (3) to MIGRATE_TO_VMX_CHECKPT (4).

  When MigrateSetState logs are present, it is likely that a vMotion operation is in progress. If you are unsure, please collect the support bundles and contact Broadcom Technical Support.

 

Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

Cause

This is caused by a very rare race condition that can occur when during vMotion process on a VVOL datastore. If a virtual machine is not actively writing to the disk and a "First Write" occurs during vMotion, this race condition may cause a CID mismatch.
"First Write" refers to the first write to disk by the guest OS after any of the following events:

• Virtual machine power-on
• vMotion
• Snapshot creation or deletion

Resolution

Broadcom VCF Support is aware of this issue. At this time, there is no workaround available.

Additional Information

VVOL datastores are currently deprecated and are planned to be removed in VCF 9.1. For more details, please refer to Deprecation of VMware vSphere Virtual Volumes (vVols) in VCF 9.0 and VVF 9.0.