HCX - Unable to Power Off Guest OS during Bulk Migration Switchover
search cancel

HCX - Unable to Power Off Guest OS during Bulk Migration Switchover

book

Article ID: 321632

calendar_today

Updated On:

Products

VMware HCX VMware Cloud on AWS

Issue/Introduction

  • This article provides information on counter measure for an issue that prevents switchover during bulk migration of HCX.
  • Bulk Migration fails at the time of cutover with the following error message.
    Error message seen in HCX UI:
    Can not poweroff VM as guest shutdown timed out and forcePowerOff is not set
  • In the app.log (/common/logs/admin/app.log), a similar error can be found:

2021-03-02 03:12:07.186 UTC [ReplicationTransferService_SvcThread-65273, Ent: HybridityAdmin, , TxId: #####-####-###-823d-94f0a4a331d3] ERROR c.v.h.s.r.j.VirtualmachineOperationJob- Job (#######-####-####-####-175b2cdda7f2) failed with exception Can not poweroff VM as guest shutdown timed out and forcePowerOff is not set java.lang.RuntimeException: Can not poweroff VM as guest shutdown timed out and forcePowerOff is not set         at com.vmware.hybridity.service.replicationtransfer.jobs.VirtualmachineOperationJob.doWaitForPowerOffVm(VirtualmachineOperationJob.java:247)         at com.vmware.hybridity.service.replicationtransfer.jobs.VirtualmachineOperationJob.run(VirtualmachineOperationJob.java:59)         at com.vmware.vchs.hybridity.messaging.LoggingJobWrapper.run(LoggingJobWrapper.java:41

Note:  Dates, Times and, UUIDs in the log line above are examples. 

Environment

VMware HCX

Cause

This issue happens when a guest OS does not reach a fully powered off state within 100 seconds of the initial "Shutdown Guest OS" call as part of bulk migration cutover.   

Resolution

Without "Force Power Off" selected, VMware HCX Bulk Migration utilizes VMware Tools to gracefully shutdown a guest OS.  As part of this mechanism, HCX will wait 100 seconds for the guest OS to be completely powered down.  If the 100 second period elapses and the virtual machine does not reflect a powered off status, the migration will fail.
If VMware Tools is not detected on the guest OS, pre-migration validation will prompt a user to select "Force Power Off".

With "Force Power Off" selected, VMware HCX Bulk Migration still utilizes VMware Tools (if present) to gracefully shutdown a guest OS. After 100 seconds has elapsed since the initial "Shutdown Guest OS" call, an immediate power off will be performed (if necessary) on the virtual machine in order to move forward with the migration.  

From HCX 4.6.1 release onwards, if the virtual machine does not power off after 100 seconds during the Bulk Migration cut-over stage, HCX waits for an additional 100 seconds to handle any delay in synchronizing with the vCenter Server before failing the migration workflow. Refer to HCX Release Notes.

Additional Information

Powered by Wolken Software