After upgrading vSphere to 8.x or higher, VMs cannot be powered off or cloned due to unsupported VMware Tools.
search cancel

After upgrading vSphere to 8.x or higher, VMs cannot be powered off or cloned due to unsupported VMware Tools.

book

Article ID: 415702

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

Post-upgrade from vSphere 7.x to 8.x, power-off and cloning operations fail on some virtual machines:

  • A general system error occurred: Undeclared fault Cause: Failed to power off the virtual machine.

  • An error occurred while quiescing the virtual machine. See the virtual machine's event log for details. An error occurred while taking a snapshot: Failed to quiesce the virtual machine. An error occurred while saving the snapshot: Failed to quiesce the virtual machine.

  • The VM's summary tab is showing VMware Tools Not running:



  •  vmware.log:
    YYYY-MM-DDThh:SS:52.839Z In(05) vmx - MKSVMX: Vigor requested a screenshot
    YYYY-MM-DDThh:SS:52.839Z In(05) svga - MKSScreenShotMgr: Taking a screenshot
    YYYY-MM-DDThh:SS.697Z In(05) vmx - MKSVMX: Vigor requested a screenshot
    YYYY-MM-DDThh:SS.697Z In(05) svga - MKSScreenShotMgr: Taking a screenshot
    YYYY-MM-DDThh:SS:07.798Z In(05) vmx - VigorTransportProcessClientPayload: opID=mg0mou06-285342-auto-6467-h5:70037339-65-01-a9-af74 seq=106876: Receiving Snapshot. Take request.
    YYYY-MM-DDThh:SS:07.802Z In(05) vmx mg0mou06-285342-auto-6467-h5:70037339-65-01-a9-af74 SnapshotVMX_TakeSnapshot start: 'clone-temp-SS28192830534', deviceState=0, lazy=0, quiesced=1, forceNative=0, tryNative=1, saveAllocMaps=0
    YYYY-MM-DDThh:SS:07.802Z In(05) vmx mg0mou06-285342-auto-6467-h5:70037339-65-01-a9-af74 4236104:VVOLLIB : VVolLib_IsVectoredSnapSupported:14372: VVolLib_IsVectoredSnapSupported: vvolVectoredSnapEnabled: 1
    YYYY-MM-DDThh:SS:07.802Z In(05)+ vmx mg0mou06-285342-auto-6467-h5:70037339-65-01-a9-af74
    YYYY-MM-DDThh:SS:07.802Z In(05) vmx mg0mou06-285342-auto-6467-h5:70037339-65-01-a9-af74 VVolObjNotifySnapshotPrepare: Changing state to VEC_SNAP_PREPARE_IN_PROG.
    YYYY-MM-DDThh:SS:07.802Z In(05) vmx mg0mou06-285342-auto-6467-h5:70037339-65-01-a9-af74 OBJLIB-VVOLOBJ :VVolObjNotifySnapshotPrepare: Building in-mem cache
    YYYY-MM-DDThh:SS:07.808Z Er(02) vmx mg0mou06-285342-auto-6467-h5:70037339-65-01-a9-af74 DISKLIB-LIB_CREATE   : DiskLib_NotifySnapshotPrepare: FiltLib failed to notify snapshot prepare: Operation completes asynchronously
    YYYY-MM-DDThh:SS:07.826Z In(05) vmx mg0mou06-285342-auto-6467-h5:70037339-65-01-a9-af74 DISKLIB-CBT   :ChangeTrackerESX_CreateMirror: Created mirror node /vmfs/devices/svm/1a9c1e9-85f95SS-cbtmirror.
    YYYY-MM-DDThh:SS:07.927Z In(05) vmx mg0mou06-285342-auto-6467-h5:70037339-65-01-a9-af74 SNAPSHOT: SnapshotPrepareTakeDoneCB: Prepare phase complete (The operation completed successfully).
    YYYY-MM-DDThh:SS:SS.538Z In(05) vmx - GuestRpcSendTimedOut: message to toolbox timed out.
    YYYY-MM-DDThh:SS:SS.538Z In(05) vmx - Tools: [AppStatus] Last heartbeat value SS8462 (last received 4s ago)
    YYYY-MM-DDThh:SS:SS.538Z In(05) vmx - TOOLS: appName=toolbox, oldStatus=1, status=2, guestInitiated=0.
    YYYY-MM-DDThh:SS:29.924Z In(05) vcpu-0 - Tools: Tools heartbeat timeout.
    YYYY-MM-DDThh:SS:29.924Z In(05) vcpu-0 - Tools: Running status rpc handler: 1 => 0.
    YYYY-MM-DDThh:SS:29.924Z In(05) vcpu-0 - Tools: Changing running status: 1 => 0.
    YYYY-MM-DDThh:SS:29.924Z In(05) vcpu-0 - Tools: [RunningStatus] Last heartbeat value SS8462 (last received 21s ago)
    YYYY-MM-DDThh:SS:29.924Z In(05) vcpu-0 - TOOLS hostVerifiedSamlToken capability set to FALSE.
    YYYY-MM-DDThh:SS:33.539Z In(05) vmx - GuestRpcSendTimedOut: message to toolbox timed out.
    YYYY-MM-DDThh:SS:37.931Z In(05) vmx mg0mou06-285342-auto-6467-h5:70037339-65-01-a9-af74 GuestRpcSendTimedOut: message to toolbox timed out.
    YYYY-MM-DDThh:SS:37.932Z In(05) vmx mg0mou06-285342-auto-6467-h5:70037339-65-01-a9-af74 Msg_Post: Warning
    YYYY-MM-DDThh:SS:37.932Z In(05) vmx mg0mou06-285342-auto-6467-h5:70037339-65-01-a9-af74 [msg.snapshot.quiesce.rpc_timeout] A timeout occurred while communicating with VMware Tools in the virtual machine.
    YYYY-MM-DDThh:SS:37.932Z In(05) vmx mg0mou06-285342-auto-6467-h5:70037339-65-01-a9-af74 ----------------------------------------
    YYYY-MM-DDThh:SS:37.933Z In(05) vmx - ToolsBackup: changing quiesce state: IDLE -> DONE
    YYYY-MM-DDThh:SS:37.933Z In(05) vmx - SnapshotVMXTakeSnapshotComplete: Done with snapshot 'clone-temp-SS28192830534': 0
    YYYY-MM-DDThh:SS:37.933Z In(05) vmx - DISKLIB-CBT   : ChangeTrackerESX_DestroyMirror: Destroyed mirror node 1a9c1e9-85f95SS-cbtmirror. SrcFd: /vmfs/volumes/xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx/VM_Name/VM_Name-ctk.vmdk, DestFd: /vmfs/volumes/xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx/VM_Name/VM_Name-ctk-mirror.vmdk.
    YYYY-MM-DDThh:SS:37.947Z In(05) vmx - SnapshotVMXTakeSnapshotComplete: Snapshot 0 failed: Failed to quiesce the virtual machine (29).
    YYYY-MM-DDThh:SS:37.947Z In(05) vmx - VVolObjNotifySnapshotDone: isEnabled: 1
    YYYY-MM-DDThh:SS:37.947Z In(05) vmx - VigorTransport_ServerSendResponse opID=mg0mou06-285342-auto-6467-h5:70037339-65-01-a9-af74 seq=106876: Completed Snapshot. Take request with messages in 30149577 US.
    YYYY-MM-DDThh:SS:39.642Z In(05) vmx - MKSVMX: Vigor requested a screenshot
    YYYY-MM-DDThh:SS:39.642Z In(05) svga - MKSScreenShotMgr: Taking a screenshot
    YYYY-MM-DDThh:SS:53.540Z In(05) vmx - GuestRpcSendTimedOut: message to toolbox timed out.
    YYYY-MM-DDThh:SS:59.648Z In(05) vmx - MKSVMX: Vigor requested a screenshot
    YYYY-MM-DDThh:SS:59.648Z In(05) svga - MKSScreenShotMgr: Taking a screenshot
    YYYY-MM-DDThh:SS:SS.541Z In(05) vmx - GuestRpcSendTimedOut: message to toolbox timed out.
    YYYY-MM-DDThh:SS:33.543Z In(05) vmx - GuestRpcSendTimedOut: message to toolbox timed out.
    YYYY-MM-DDThh:SS:53.545Z In(05) vmx - GuestRpcSendTimedOut: message to toolbox timed out.

     

Environment

VVF 9.x
vSphere 8.x

Cause

The problem occurs because VMware Tools version 10.1.5 (released February 23, 2017) is incompatible with vSphere 8.0 and later versions.

Resolution

Upgrade VMware Tools to a version that supports vSphere 8.0 and later releases.

Product Interoperability Matrix
https://interopmatrix.broadcom.com/Interoperability?col=1,&row=139

Additional Information

Powered by Wolken Software