VM Powers Off During Backup Due to Content ID (CID) Mismatch on Physical RDM
search cancel

VM Powers Off During Backup Due to Content ID (CID) Mismatch on Physical RDM

book

Article ID: 447114

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms

  • A production virtual machine (VM) unexpectedly powers off or is restarted by vSphere HA.
  • The issue occurs specifically during the window of a third-party backup (e.g., Commvault) snapshot operation.
  • The VM /vmfs/volumes/DatastoreName/VM folder/vmware.log contains errors similar to:
   DISK:bora/devices/disk/disk.c:7490 contentID mismatch on 'scsiX:Y' - was 0xXXXXXXXX, is 0xYYYYYYYY
   [msg.disk.notCheckpointed3] Resuming virtual disk 'scsiX:Y' failed. The disk has been modified since a snapshot was taken...
   [msg.checkpoint.continuesync.error] An operation required the virtual machine to quiesce and the virtual machine was unable to continue running.

Environment

  • VMware vSphere ESXi 7.x 
  • VMware vSphere ESXi 8.x 

Cause

This issue is caused by an external modification of the physical LUN's metadata during the microscopic window when VMware temporarily "closes" the disk descriptor to freeze the VM state for a snapshot (the stun/unstun cycle).

While independent-persistent disks do not create redo logs (snapshots), they still undergo the following sequence:

  1. Stun: Backup requests a snapshot.
  2. Device Teardown: VMware temporarily removes the virtual device.
  3. Disk Closure: VMware closes the lock on the physical disk descriptor.
  4. The Re-Open: VMware attempts to reopen the disk to the base file.

If the underlying SAN storage (e.g., hardware-level snapshot, replication, or an in-flight write completion) modifies the LUN's Content ID (CID) between Step 3 and Step 4, VMware detects the mismatch. To protect the database/file system from potential corruption, VMware fails the sync and powers off the VM.

Resolution

  1. SAN Log Analysis: Review storage logs for the specific LUN at the exact failure timestamp.
  2. Hardware Snapshot Check: Confirm if the SAN triggered a hardware-level task (clone/replication) at the same second.
  3. Backup Job Correlation: Verify if backup software was running high-intensity tasks like "Synthetic Full".
  4. Manual CID Alignment: If the VM fails to power on, manually align the contentID in the .vmdk descriptor with the ID reported in logs. Refer KB :Consolidating the VM disk fails with CID_MISMATCH (301128)

Additional Information

Related Information

Powered by Wolken Software