Quiesced snapshots of Windows 2008R2/2012/2012R2 VMs with VMware Tools 10.1.0/10.1.5 fails

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Virtual Machine tools status changes automatically from running to not running.
• Quiesced snapshots of Windows 2008R2/2012/2012R2 VMs with VMware Tools 10.1.0 or 10.1.5 are sporadically failing due to tools service not responding.
• Debug logging is been enabled for VMware Tools within Guest OS.
• In the vmsvc.log you will see entries similar to:

[ warning] [vmsvc] ----Win32 exception detected, exceptionCode 0xc0000005 (access violation)----

[ warning] [vmsvc] ExceptionAddress 0x7fef1be4cef eflags 0x00010202
[ warning] [vmsvc] rwFlags 0 badAddr 0x28
[ warning] [vmsvc] rax 0x1 rbx 0 rcx 0x4d9eb3c9e5d60000
[ warning] [vmsvc] rdx 0x1 rsi 0 rdi 0x980540
[ warning] [vmsvc] r8 0x980570 r9 0x3 r10 0x2
[ warning] [vmsvc] r11 0x693c5e000a001d r12 0x127f790 r13 0
[ warning] [vmsvc] r14 0 r15 0
[ warning] [vmsvc] rip 0x7fef1be4cef rsp 0x127f600 rbp 0x18f810
[ warning] [vmsvc] LastBranchToRip 0 LastBranchFromRip 0
[ warning] [vmsvc] LastExceptionToRip 0 LastExceptionFromRip 0
[ warning] [vmsvc] The following data was delivered with the exception:[ warning] [vmsvc] -- 0
[ warning] [vmsvc] -- 0x28
[ info] [vmsvc] CoreDump: Minidump file C:\Windows\TEMP\vmware-SYSTEM\vmtoolsd.dmp exists. Rotating ...
[ warning] [vmsvc] CoreDump: Writing minidump to C:\Windows\TEMP\vmware-SYSTEM\vmtoolsd.dmp
[ info] [vmsvc] CoreDump: including module base 0x0x13ff70000 size 0x0x00014000

 

• In the vmware.log, you see entries similar to:

vmx| I120: SnapshotVMX_TakeSnapshot start: '<backup name>-148911122268886c5d233d46c6169ae283395bd7bdf6b6af5e', deviceState=0, lazy=0, logging=0, quiesced=1, forceNative=0, tryNative=1, sibling=0 saveAllocMaps=0 cb=1D2B96E0, cbData=32574D80
vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.
vmx| I120: Tools: Tools heartbeat timeout.
vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.
vmx| I120: Msg_Post: Warning
vmx| I120: [msg.snapshot.quiesce.rpc_timeout] A timeout occurred while communicating with VMware Tools in the virtual machine.
| vmx| I120: ----------------------------------------
vmx| I120: ToolsBackup: changing quiesce state: IDLE -> DONE
vmx| I120: SnapshotVMXTakeSnapshotComplete: Done with snapshot 'Avamar-148911122268886c5d233d46c6169ae283395bd7bdf6b6af5e': 0
vmx| I120: SnapshotVMXTakeSnapshotComplete: Snapshot 0 failed: Failed to quiesce the virtual machine (40)

vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.

vmx| I120: Tools: Tools heartbeat timeout.

vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.

vmx| I120: Msg_Post: Warning

vmx| I120: [msg.snapshot.quiesce.rpc_timeout] A timeout occurred while communicating with VMware Tools in the virtual machine.

| vmx| I120: ----------------------------------------

vmx| I120: ToolsBackup: changing quiesce state: IDLE -> DONE

vmx| I120: SnapshotVMXTakeSnapshotComplete: Done with snapshot 'Avamar-148911122268886c5d233d46c6169ae283395bd7bdf6b6af5e': 0

vmx| I120: SnapshotVMXTakeSnapshotComplete: Snapshot 0 failed: Failed to quiesce the virtual machine (40)

 

 

 

Cause

If you disable Volume Shadow Copy(VSS) support by not installing VMware snapshot provider this may lead to VSS provider can't initialize and lead to a invalid class pointer access during complete initialization.

Resolution

This issue is resolved in VMware tools version 10.1.10, available at VMware Downloads.

Workaround:

To workaround this issue:

• Uninstall VMware tools and reinstall VMware tools with VSS provider.
• If you need to work without VSS support, disable VSS support by navigating to C:\ProgramData\VMware\VMwareTools\Tools.conf and add entries:
  
  [vmbackup]
  enableVSS=false

Additional Information

Enabling debug logging for VMware Tools within a guest operating system