Bulk Migration may fail due to "Invalid Power State" of VM

Products

• VMware HCX

Issue/Introduction

• Bulk Migration may fail during the switchover process with the following errors:
  Power off task failed due to(vim.fault.InvalidPowerState) { faultCause = null, faultMessage = null, requestedState = poweredOn, existingState = poweredOff }

Sync operation couldn't be performed as Virtual Machine <vm-id> is not in powered off state
• The following error is observed in /common/logs/admin/app.log:
  <timestamps> UTC [ReplicationTransferService_SvcThread-36921, Ent: HybridityAdmin, , TxId: <txID>] INFO  c.v.h.s.r.j.VirtualmachineOperationJob- waiting for shutdown
<timestamps> UTC [ReplicationTransferService_SvcThread-36921, Ent: HybridityAdmin, , TxId: <txID>] INFO  c.v.h.s.r.j.VirtualmachineOperationJob- Waiting for guest shutdown, Retry count: 1
<timestamps> UTC [ReplicationTransferService_SvcThread-36922, Ent: HybridityAdmin, , TxId: <txID>] INFO  c.v.h.s.r.j.VirtualmachineOperationJob- Waiting for guest shutdown, Retry count: 2
<timestamps> UTC [ReplicationTransferService_SvcThread-36914, Ent: HybridityAdmin, , TxId: <txID>] INFO  c.v.h.s.r.j.VirtualmachineOperationJob- Waiting for guest shutdown, Retry count: 3
<timestamps> UTC [ReplicationTransferService_SvcThread-36930, Ent: HybridityAdmin, , TxId: <txID>] INFO  c.v.h.s.r.j.VirtualmachineOperationJob- Waiting for guest shutdown, Retry count: 4
<timestamps> UTC [ReplicationTransferService_SvcThread-36914, Ent: HybridityAdmin, , TxId: <txID>] INFO  c.v.h.s.r.j.VirtualmachineOperationJob- Waiting for guest shutdown, Retry count: 5
<timestamps> UTC [ReplicationTransferService_SvcThread-36931, Ent: HybridityAdmin, , TxId: <txID>] ERROR c.v.h.s.r.j.VirtualmachineOperationJob- Job (########-####-####-####-##########a2) failed with exception Power off task failed due to(vim.fault.InvalidPowerState) {
   faultCause = null,
   faultMessage = null,
   requestedState = poweredOn,
   existingState = poweredOff
}
java.lang.RuntimeException: Power off task failed due to(vim.fault.InvalidPowerState) {
   faultCause = null,
   faultMessage = null,
   requestedState = poweredOn,
   existingState = poweredOff
}

Cause

1. During the Bulk Migration switchover, the Graceful Power Off task did not complete, even with the 'forcePowerOff' option enabled, because the VM had already powered off due to a delayed shutdown. This caused the migration to fail.
   
   
2. The guest OS shutdown took more than 100 seconds (20 seconds per retry attempt). As a result of this timing issue, when HCX initiated the power-off task, the guest OS shutdown task was also completing at the same time. Subsequently, vCenter threw an error, stating it was unable to power off an already powered-off VM.

An additional cause of not being able to Power Off Guest OS during Switchover Using HCX Bulk Migration when 'forcePowerOff' is not set is well described here: HCX - Unable to Power Off Guest OS during Bulk Migration Switchover

•  

Resolution

Workaround:

• In the event that other VMs take longer or cannot be shutdown gracefully from Guest OS, the recommendation is for the customer to enable "Force Power Off" upon scheduling the migrations.
• It is also recommended to use "Seed Checkpoint" for Bulk Migrations which is available from HCX 4.1.0 release onwards, in the event that the workflow fails and rolls back at the cut over stage, so when rescheduling the migrations, the workflow will try to reuse the data already copied in the previous attempt.

Note: The recommendation is to not perform cleanup operation of failed job which leads to the removal of seed data.

 

Additional Information

It only impacts HCX Bulk migration workflow. There is no impact to other migration profiles like vMotion/RAV/Cold Migration.