Quiesced snapshot of Windows Server virtual machine fails. Error 'VssSyncStart' operation failed: IDispatch error #8454 (0x80042306) seen in vmware.log

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

Running a quiesced snapshot (or running a backup job with quiescing) fails with the following errors on Server VM's

• In the vCenter UI, you see the error: 

An error occurred while quiescing the virtual machine.  See the virtual machine's event log for details. 

• The vmware.log related to the impacted virtual machine reports error: 

YYYY-MM-DDTHH:MM:SSZ| vmx| I125: SNAPSHOT: SnapshotPrepareTakeDoneCB: Prepare phase complete (The operation completed successfully).
YYYY-MM-DDTHH:MM:SSZ| vcpu-1| I125: ToolsBackup: changing quiesce state: IDLE -> STARTED
YYYY-MM-DDTHH:MM:SSZ| vmx| I125: Msg_Post: Warning
YYYY-MM-DDTHH:MM:SSZ| vmx| I125: [msg.snapshot.quiesce.vmerr] The guest OS has reported an error during quiescing.
YYYY-MM-DDTHH:MM:SSZ| vmx| I125+ The error code was: 5
YYYY-MM-DDTHH:MM:SSZ| vmx| I125+ The error message was: 'VssSyncStart' operation failed: IDispatch error #8454 (0x80042306)
YYYY-MM-DDTHH:MM:SSZ| vmx| I125: ----------------------------------------
YYYY-MM-DDTHH:MM:SSZ| vmx| I125: ToolsBackup: changing quiesce state: STARTED -> ERROR_WAIT
YYYY-MM-DDTHH:MM:SSZ| vmx| I125: ToolsBackup: changing quiesce state: ERROR_WAIT -> IDLE
YYYY-MM-DDTHH:MM:SSZ| vmx| I125: ToolsBackup: changing quiesce state: IDLE -> DONE
YYYY-MM-DDTHH:MM:SSZ| vmx| I125: SnapshotVMXTakeSnapshotComplete: Done with snapshot 'VM Snapshot 7%252###%25###19 HH:MM AM UTC-##:##'
YYYY-MM-DDTHH:MM:SSZ| vmx| I125: SnapshotVMXTakeSnapshotComplete: Snapshot 0 failed: Failed to quiesce the virtual machine (29).
YYYY-MM-DDTHH:MM:SSZ| vmx| I125: VigorTransport_ServerSendResponse opID=TakeSnapshotFormMediator-add-4334090-ngc:703####2-f-e6-#### seq=55743: Completed Snapshot request.
YYYY-MM-DDTHH:MM:SSZ| vmx| I125: Guest: *** WARNING: GuestInfo collection interval longer than expected; actual=73 sec, expected=30 sec. ***

• Windows Event Viewer shows ID 8193 reports:

Volume Shadow Copy Service error: Unexpected error calling routine IOCTL_DISK_SET_SNAPSHOT_INFO(\\.\PHYSICALDRIVE1) fails with winerror 1168.  hr = 0x80070490, Element not found.
Operation:
   Processing EndPrepareSnapshots
   Executing Asynchronous Operation
Context:
   Provider Name: VMware Snapshot Provider
   Provider Version: 1.0.0
   Provider ID: {564d7761-####-2056-####-2050726f7669}
   Current State: DoSnapshotSet
 
Volume Shadow Copy Service error: Unexpected error calling routine IOCTL_DISK_SET_SNAPSHOT_INFO(\\.\PHYSICALDRIVE1) fails with winerror 1168.  hr = 0x80070490, Element not found.
Operation:
   Processing AbortSnapshots
   Executing Asynchronous Operation
Context:
   Device: \\.\PHYSICALDRIVE1
   Provider Name: VMware Snapshot Provider
   Provider Version: 1.0.0
   Provider ID: {564d7761-####-2056-####-2050726f7669}
   Current State: DoSnapshotSet

Environment

VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0

Cause

This issue occurs because the VM has a GPT disk without an MSR (Microsoft Reserved) partition. VMware Snapshot Provider works under Microsoft VSS framework.  The Microsoft VSS framework will send the IOCTL IOCTL_DISK_SET_SNAPSHOT_INFO to the disks. If the disk is a GPT disk without a MSR partition, the IOCTL would fail and whole VSS process fails.  Microsoft requires every GPT disk to have an MSR: 
 
Windows and GPT FAQ

Resolution

To resolve this issue, use one of these options:

• Option 1: 

Add a new GPT disk with an MSR and migrate the data to the new disk.  Then, remove the old GPT disk.  Microsoft support may need to verify the exact procedure, depending on what is stored on the disk. 

• Option 2: 

If there is no space available to create a new MSR, shrink the volume by the default MSR size (32MB or 128MB).  Microsoft support may need to verify the exact procedure for this.

Workaround:

To work around this issue, use one of these options:

Note: These options will prohibit application consistent backups, if there are no available 3rd party VSS Snapshot providers.

• Option 1:
  Do not use the "VMWare Snapshot Provider" service by disabling it or uninstalling that component from VMware Tools.  
  
  
• Option 2:
  Do not quiesce the guest OS when taking snapshots (or backups) or disable application level quiescing. For more information, see Disabling VSS quiesced application based snapshots in virtual machine.
  
  
• Option 3:
  Disable the pre-backup quiesce call to the vSphere Environment. See backup vendor specific documentation on steps.