Quiesced snapshots of Windows 2008R2/2012/2012R2 VMs with VMware Tools 10.1.0/10.1.5 fails

Products

• VMware vSphere ESXi

Issue/Introduction

• Virtual Machine tools status changes automatically from running to not running.
• Quiesced snapshots of Windows 2008R2/2012/2012R2 VMs with VMware Tools 10.1.0 or 10.1.5 are sporadically failing due to tools service not responding.
• Debug logging has been enabled for VMware Tools within Guest OS.
• In the vmsvc.log,there will be similar entries as shown below:
  
  [ warning] [vmsvc] ----Win32 exception detected, exceptionCode 0xc0000005 (access violation)----
  [ warning] [vmsvc] ExceptionAddress 0x7fef1be4cef eflags 0x00010202
  [ warning] [vmsvc] rwFlags 0 badAddr 0x28
  [ warning] [vmsvc] rax 0x1 rbx 0 rcx 0x4d9eb3c9e5d60000
  [ warning] [vmsvc] rdx 0x1 rsi 0 rdi 0x980540
  [ warning] [vmsvc] r8 0x980570 r9 0x3 r10 0x2
  [ warning] [vmsvc] r11 0x693c5e000a001d r12 0x127f790 r13 0
  [ warning] [vmsvc] r14 0 r15 0
  [ warning] [vmsvc] rip 0x7fef1be4cef rsp 0x127f600 rbp 0x18f810
  [ warning] [vmsvc] LastBranchToRip 0 LastBranchFromRip 0
  [ warning] [vmsvc] LastExceptionToRip 0 LastExceptionFromRip 0
  [ warning] [vmsvc] The following data was delivered with the exception:[ warning] [vmsvc] -- 0
  [ warning] [vmsvc] -- 0x28
  [ info] [vmsvc] CoreDump: Minidump file C:\Windows\TEMP\vmware-SYSTEM\vmtoolsd.dmp exists. Rotating ...
  [ warning] [vmsvc] CoreDump: Writing minidump to C:\Windows\TEMP\vmware-SYSTEM\vmtoolsd.dmp
  [ info] [vmsvc] CoreDump: including module base 0x0x13ff70000 size 0x0x00014000

• In the vmware.log, there will be similar entries as shown below:
  
  vmx| I120: SnapshotVMX_TakeSnapshot start: '<backup name>-148911122268886c5d233d46c6169ae283395bd7bdf6b6af5e', deviceState=0, lazy=0, logging=0, quiesced=1, forceNative=0, tryNative=1, sibling=0 saveAllocMaps=0 cb=1D2B96E0, cbData=32574D80
  vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.
  vmx| I120: Tools: Tools heartbeat timeout.
  vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.
  vmx| I120: Msg_Post: Warning
  vmx| I120: [msg.snapshot.quiesce.rpc_timeout] A timeout occurred while communicating with VMware Tools in the virtual machine.
  | vmx| I120: ----------------------------------------
  vmx| I120: ToolsBackup: changing quiesce state: IDLE -> DONE
  vmx| I120: SnapshotVMXTakeSnapshotComplete: Done with snapshot 'Avamar-148911122268886c5d233d46c6169ae283395bd7bdf6b6af5e': 0
  vmx| I120: SnapshotVMXTakeSnapshotComplete: Snapshot 0 failed: Failed to quiesce the virtual machine (40)
  vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.
  vmx| I120: Tools: Tools heartbeat timeout.
  vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.
  vmx| I120: Msg_Post: Warning
  vmx| I120: [msg.snapshot.quiesce.rpc_timeout] A timeout occurred while communicating with VMware Tools in the virtual machine.
  | vmx| I120: ----------------------------------------
  vmx| I120: ToolsBackup: changing quiesce state: IDLE -> DONE
  vmx| I120: SnapshotVMXTakeSnapshotComplete: Done with snapshot 'Avamar-148911122268886c5d233d46c6169ae283395bd7bdf6b6af5e': 0
  vmx| I120: SnapshotVMXTakeSnapshotComplete: Snapshot 0 failed: Failed to quiesce the virtual machine (40)

Cause

Disable Volume Shadow Copy (VSS) support by not installing VMware snapshot provider this may result in the inability to initialize the VSS provider, which can lead to an invalid class pointer access during complete initialization.

Resolution

This issue is resolved in VMware tools version 10.1.10.

Workaround:

• Uninstall VMware Tools and reinstall VMware tools with VSS provider.
• If needed to work without VSS support, disable VSS support by navigating to C:\ProgramData\VMware\VMwareTools\Tools.conf and add entries:
  
  [vmbackup]
enableVSS=false

Additional Information

Enabling debug logging for VMware Tools within a guest operating system