[VMC on AWS][Internal] Migrating a Virtual Machine from One Host to another fails with an error: "Operation Timed out"

Products

• VMware Cloud on AWS

Issue/Introduction

This article provides steps to troubleshoot the error: "Operation Timed out" received while performing the Migration of a VM from one host to another.

Symptoms:
Migrating a Virtual Machine from One Host to another fails with an error: "Operation Timed out"


The below entries are observed  in the hostd log on the source host:

<99>1 2021-01-12T14:33:28.535907+00:00 vcenter vpxd 8371 - -  Event [7866265] [1-1] [2021-01-12T14:33:28.535348Z] [vim.event.EventEx] [info] [] [SDDC-Datacenter] [7866265] [Alarm 'Migration error' on 0006-Win7-WA-07-CSTC-CLASSROOM-CS4036PB-I-001-1 triggered by event 7866259 'Cannot migrate 0006-Win7-WA-07-CSTC-CLASSROOM-CS4036PB-I-001-1 from 10.2.2.5, WorkloadDatastore in SDDC-Datacenter to 10.2.2.4, WorkloadDatastore in SDDC-Datacenter']

<99>1 2021-01-12T14:39:27.877649+00:00 vcenter vpxd 8371 - -  Event [7866562] [1-1] [2021-01-12T14:39:27.87712Z] [vim.event.EventEx] [info] [PHILIPS-CS\I-001] [SDDC-Datacenter] [7863211] [Failed to revert the execution state of the virtual machine 0006-Win7-WA-07-CSTC-CLASSROOM-CS4036PB-I-001-1 on host 10.2.2.5, in compute resource Cluster-1 to snapshot InitialSnapshot, with ID 1]


<99>2021-01-12T14:39:27.860Z esx-1.sddc-18-195-1-152.vmwarevmc.com Hostd: warning hostd[2101667] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:4381f57d76d44192-8523bdb1381338c8/d617f25f-1240-8669-8997-0281b10c5e06/0006-Win7-WA-07-CSTC-CLASSROOM-CS4036PB-I-001-1.vmx opID=khsmvbpf-6174259-auto-3oc38-h5:71320658-f3-86-0f55 user=vpxuser:PHILIPS-CS\I-001] Failed to revert snapshot(s): N3Vim5Fault8Timedout9ExceptionE(Fault cause: vim.fault.Timedout


<99>2021-01-12T14:39:27.697Z esx-1.sddc-18-195-1-152.vmwarevmc.com Fdm: info fdm[2105324] [Originator@6876 sub=Policy opID=lro-692-31caf70-01] Reset no 0 failed for VM /vmfs/volumes/vsan:4381f57d76d44192-8523bdb1381338c8/d617f25f-1240-8669-8997-0281b10c5e06/0006-Win7-WA-07-CSTC-CLASSROOM-CS4036PB-I-001-1.vmx


<99>2021-01-12T15:14:20.482Z esx-5.sddc-18-195-1-152.vmwarevmc.com Hostd: verbose hostd[2107109] [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/vsan:4381f57d76d44192-8523bdb1381338c8/7f0ff25f-98b9-c098-2470-02d4ddd60f18/0006-Win7-WA-03-CSTC-CLASSROOM-CS4036PB-I-001-1.vmx] Get vigor fields VigorCpp::Devices::Migrate::Online translated error to vim.fault.InvalidState


<99>2021-01-12T15:23:31.806Z esx-1.sddc-18-195-1-152.vmwarevmc.com Hostd: info hostd[2101428] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:4381f57d76d44192-8523bdb1381338c8/d617f25f-1240-8669-8997-0281b10c5e06/0006-Win7-WA-07-CSTC-CLASSROOM-CS4036PB-I-001-1.vmx opID=63fb639a-e78c user=vpxuser] Error retrieving file layout : N3Vim5Fault8Timedout9ExceptionE(Fault cause: vim.fault.Timedout

Cause

Virtual Machine (VM) may have locks on its files or it could be left in a zombie state i.e. The VM is unregistered from the VMC vCenter.

Resolution

Perform the below steps to resolve the Migration Failure issue : 

1. Power off the VM
2. Consolidate All snapshots present on the VM.
3. Attempt the Migration again from the powered-off state.
4. If the Migration succeeds then ensure that the VM is powered On and attempt a test with the original Migration operation again.

Additional Information

Migrate a Virtual Machine to a New Host and Datastore by Using vMotion in the vSphere Web Client:
https://via.vmw.com/ESST

Impact/Risks:
Unable to perform any actions on the VM.