vMotion fails at 74% with the error, “The VM failed to resume on the destination during early power on”.
search cancel

vMotion fails at 74% with the error, “The VM failed to resume on the destination during early power on”.

book

Article ID: 326930

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

Symptoms:
  • In the hostd.log you see messages similar to:

YYYY-MM-DDTHH:MM:SS info hostd[A1BF1B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:12312312########-######3322332233/########-####-####-####-########1234/test-vm.vmx] Checking for all objects accessibility (VM's current state: VM_STATE_EMIGRATING, stable? false)
YYYY-MM-DDTHH:MM:SS info hostd[A1BF1B70] [Originator@6876 sub=VmObjectStorageMonitor] Object UUID ########-####-####-####-########1234 APD state: Healthy
YYYY-MM-DDTHH:MM:SS info hostd[A1BF1B70] [Originator@6876 sub=VmObjectStorageMonitor] Object UUID ########-####-####-####-########7c20 APD state: Healthy
YYYY-MM-DDTHH:MM:SS info hostd[A1BF1B70] [Originator@6876 sub=Vcsvc.VMotionSrc (5422550518507384986)] ResolveCb: VMX reports needsUnregister = false for migrateType MIGRATE_TYPE_VMOTION
YYYY-MM-DDTHH:MM:SS info hostd[A1BF1B70] [Originator@6876 sub=Vcsvc.VMotionSrc (5422550518507384986)] ResolveCb: Failed with fault: (vim.fault.GenericVmConfigFault) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "msg.migrate.fail.dst",
-->          message = "The source detected that the destination failed to resume.",
-->       }
-->    ],
-->    reason = "The source detected that the destination failed to resume.",
-->    messageInfo = (vim.vm.Message) [
-->       (vim.vm.Message) {
-->          id = "msg.migrate.fail.dst",
-->          text = "The source detected that the destination failed to resume.",
-->       }
-->    ]
-->    msg = "The source detected that the destination failed to resume.
--> "
--> }
YYYY-MM-DDTHH:MM:SS info hostd[A1BF1B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:12312312########-######3322332233/########-####-####-####-########1234/test-vm.vmx] State Transition (VM_STATE_EMIGRATING -> VM_STATE_ON)
In the vmkernel.log you see messages similar to:
YYYY-MM-DDTHH:MM:SS cpu13:857334255)WARNING: Migrate: 5454: 5422550532447652106 S: Migration considered a failure by the VMX.  It is most likely a timeout, but check the VMX log for the true error.
YYYY-MM-DDTHH:MM:SS cpu17:420159321)WARNING: Migrate: 5454: 5422550533132847308 S: Migration considered a failure by the VMX.  It is most likely a timeout, but check the VMX log for the true error.
YYYY-MM-DDTHH:MM:SS cpu28:946706621)WARNING: Migrate: 5454: 5422550533291405060 S: Migration considered a failure by the VMX.  It is most likely a timeout, but check the [root@hostname:/vmfs/volumes/vsan:12312312########-######3322332233/########-####-####-####-########1234] 
In the vmware.log of test-vm VM you see messages similar to:

YYYY-MM-DDTHH:MM:SS| vmx| I125: Msg_Post: Error
YYYY-MM-DDTHH:MM:SS| vmx| I125: [msg.nvram.wback1] Could not create '/vmfs/volumes/vsan:12312312########-######3322332233/########-####-####-####-########1234/test-vm.nv
ram': Failed to lock the file
YYYY-MM-DDTHH:MM:SS| vmx| I125: ----------------------------------------
YYYY-MM-DDTHH:MM:SS| vmx| I125: Vigor_MessageRevoke: message 'msg.nvram.wback1' (seq 47059135) is revoked
YYYY-MM-DDTHH:MM:SS| vmx| W115: NVRAMMGR: Failed to create default NVRAM file (test-vm.nvram)
YYYY-MM-DDTHH:MM:SS| vmx| I125: Module Nvman power on failed.
YYYY-MM-DDTHH:MM:SS| vmx| I125: VMX_PowerOn: ModuleTable_PowerOn = 0
YYYY-MM-DDTHH:MM:SS| vmx| I125: MigrateSetStateFinished: type=2 new state=12
YYYY-MM-DDTHH:MM:SS| vmx| I125: MigrateSetState: Transitioning from state 11 to 12.
YYYY-MM-DDTHH:MM:SS| vmx| I125: Migrate: Caching migration error message list:
YYYY-MM-DDTHH:MM:SS| vmx| I125: [msg.migrate.resume.fail] The VM failed to resume on the destination during early power on.
YYYY-MM-DDTHH:MM:SS| vmx| I125: Msg_Post: Error
YYYY-MM-DDTHH:MM:SS| vmx| I125: [msg.migrate.resume.fail] The VM failed to resume on the destination during early power on.
YYYY-MM-DDTHH:MM:SS| vmx| I125: ----------------------------------------
YYYY-MM-DDTHH:MM:SS| vmx| I125: Vigor_MessageRevoke: message 'msg.migrate.resume.fail' (seq 47059136) is revoked
YYYY-MM-DDTHH:MM:SS| vmx| I125: ToolsISO: Selected Tools ISO 'windows.iso' for 'windows7srv-64' guest.
YYYY-MM-DDTHH:MM:SS| vmx| I125: Tools: ToolsRunningStatus_Exit, delayedRequest is 0x0
YYYY-MM-DDTHH:MM:SS| vmx| I125: VMXVmdb_SetToolsVersionStatus: status value set to 'ok', 'current', install possible
YYYY-MM-DDTHH:MM:SS| vmx| I125: SVMotion_PowerOff: Not running Storage vMotion. Nothing to do
YYYY-MM-DDTHH:MM:SS| svga| I125: SVGA thread is alive
YYYY-MM-DDTHH:MM:SS| svga| I125: SVGA thread is exiting
YYYY-MM-DDTHH:MM:SS| vmx| I125: Destroying virtual dev for scsi0:0 vscsi=10775
YYYY-MM-DDTHH:MM:SS| vmx| I125: VMMon_VSCSIStopVports: No such target on adapter
YYYY-MM-DDTHH:MM:SS| mks| I125: MKS PowerOff
YYYY-MM-DDTHH:MM:SS| mks| I125: MKS thread is exiting
YYYY-MM-DDTHH:MM:SS| vmx| I125: Migrate: cleaning up migration state.



Environment

VMware vSAN (All Versions)

Cause

The vMotion failed due to nvram lock not being released. This happens due to the host not being able to release the lock on NVRAM - Capacity disk on which VM is running. It can be due to an older version of firmware on NVRAM or older controller driver firmware or both.

Resolution



Workaround:
  • Shutdown / Power off VM and then power on the VM to clear the nvram lock then vMotion should work fine. 
  • If the above steps does not resolve the issue then you would need to refer the below KB to investigate if there is any other lock on virtual machines. 
For more details, refer to KB Article Investigating Virtual Machine file locks on ESXi Host(s)


Powered by Wolken Software