HCX - Bulk Migration may fail due to "Invalid Power State" of VM

Products

• VMware HCX

Issue/Introduction

• Bulk Migration may fail during the switchover process with the following error:
  Power off task failed due to(vim.fault.InvalidPowerState) { faultCause = null, faultMessage = null, requestedState = poweredOn, existingState = poweredOff }
  
  
• The following error is observed in /common/logs/admin/app.log:
  <timestamps> UTC [ReplicationTransferService_SvcThread-36921, Ent: HybridityAdmin, , TxId: <txID>] INFO  c.v.h.s.r.j.VirtualmachineOperationJob- waiting for shutdown
<timestamps> UTC [ReplicationTransferService_SvcThread-36921, Ent: HybridityAdmin, , TxId: <txID>] INFO  c.v.h.s.r.j.VirtualmachineOperationJob- Waiting for guest shutdown, Retry count: 1
<timestamps> UTC [ReplicationTransferService_SvcThread-36922, Ent: HybridityAdmin, , TxId: <txID>] INFO  c.v.h.s.r.j.VirtualmachineOperationJob- Waiting for guest shutdown, Retry count: 2
<timestamps> UTC [ReplicationTransferService_SvcThread-36914, Ent: HybridityAdmin, , TxId: <txID>] INFO  c.v.h.s.r.j.VirtualmachineOperationJob- Waiting for guest shutdown, Retry count: 3
<timestamps> UTC [ReplicationTransferService_SvcThread-36930, Ent: HybridityAdmin, , TxId: <txID>] INFO  c.v.h.s.r.j.VirtualmachineOperationJob- Waiting for guest shutdown, Retry count: 4
<timestamps> UTC [ReplicationTransferService_SvcThread-36914, Ent: HybridityAdmin, , TxId: <txID>] INFO  c.v.h.s.r.j.VirtualmachineOperationJob- Waiting for guest shutdown, Retry count: 5
<timestamps> UTC [ReplicationTransferService_SvcThread-36931, Ent: HybridityAdmin, , TxId: <txID>] ERROR c.v.h.s.r.j.VirtualmachineOperationJob- Job (########-####-####-####-##########a2) failed with exception Power off task failed due to(vim.fault.InvalidPowerState) {
   faultCause = null,
   faultMessage = null,
   requestedState = poweredOn,
   existingState = poweredOff
}
java.lang.RuntimeException: Power off task failed due to(vim.fault.InvalidPowerState) {
   faultCause = null,
   faultMessage = null,
   requestedState = poweredOn,
   existingState = poweredOff
}

Cause

1. During the Bulk Migration switchover, the Graceful Power Off task did not complete, even with the 'forcePowerOff' option enabled, because the VM had already powered off due to a delayed shutdown. This caused the migration to fail.
   
   
2. The guest OS shutdown took more than 100 seconds (20 seconds per retry attempt). As a result of this timing issue, when HCX initiated the power-off task, the guest OS shutdown task was also completing at the same time. Subsequently, vCenter threw an error, stating it was unable to power off an already powered-off VM.

An additional cause of not being able to Power Off Guest OS during Switchover Using HCX Bulk Migration when 'forcePowerOff' is not set is well described here: HCX - Unable to Power Off Guest OS during Bulk Migration Switchover

•  

Resolution

From HCX 4.6.1 release onwards, if the virtual machine does not power off after 100 seconds during the Bulk Migration cut-over stage, HCX waits for an additional 100 seconds to handle any delay in synchronizing with the vCenter Server before failing the migration workflow. 

Workaround:

• In the event that other VMs take longer or cannot be shutdown gracefully from Guest OS, the recommendation is for the customer to enable "Force Power Off" upon scheduling the migrations.
• It is also recommended for the customer to use "Seed Checkpoint" for Bulk Migrations which is available from HCX 4.1.0 release onwards, in the event that the workflow fails and rolls back at the cut over stage, so when rescheduling the migrations, the workflow will try to reuse the data already copied in the previous attempt.
  Note: The recommendation is to not perform cleanup operation of failed job which leads to the removal of seed data.

 

Additional Information

It only impacts HCX Bulk migration workflow. There is no impact to other migration profiles like vMotion/RAV/Cold Migration.