Bulk migration switchover failed : "Sync operation couldn't be performed as Virtual Machine 'vm-####' is not in powered off state"

Products

• VMware HCX

Issue/Introduction

• Bulk Migration fails at the time of cutover with the following error message.
  
  
  • Error message seen in HCX UI:
    "Sync operation couldn't be performed as Virtual Machine 'vm-####' is not in powered off state"
    
    
    
    
  • Log Entry: In the /common/logs/admin/app.log file, a similar error can be found
    

    ERROR c.v.h.s.r.jobs.SyncOperationJob- Sync operation couldn't be performed as Virtual Machine 'vm-####' is not in powered off state
    <timestamps> UTC [ReplicationTransferService_SvcThread-7250, Ent: HybridityAdmin, , TxId: #######-####-####-####-725417890c75] ERROR c.v.h.s.r.jobs.SyncOperationJob- EventCode:{}. Pre-requisite for sync operation are not met. Reason: 11061
    <timestamps> UTC [ReplicationTransferService_SvcThread-7248, Ent: HybridityAdmin, , TxId: #######-####-####-####-725417890c75] INFO c.v.h.s.r.j.ReplicationSwitchoverJob- Switchover #######-####-####-####-62dcf3501178 running in state WAIT_FOR_OFFLINE_SYNC

    
    
    

Environment

VMware HCX 4.11.3

Cause

This issue can occur when there is a difference between HCX and vCenter regarding the powered-on state of a VM.

Resolution

• This issue is resolved in VMware HCX 4.11.4 and later, available at Broadcom downloads.

• If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

• If you believe you have encountered this issue and are unable to upgrade, open a support case with Broadcom Support and refer to this KB article.
  For more information, see Creating and managing Broadcom support cases.

Workaround:

• Retry the failed migration.
• If you are currently running HCX 4.11.3 and cannot upgrade, follow these steps to use the provided script, which allows extra time for the vCenter to fully stop the VM service.

Pre-requisites:

• Ensure there are no active migrations or tasks running in HCX before proceeding.
• Important: The script must be run on both the source (Connector) and destination (Cloud) HCX managers.

Script usage:

Steps:

1. Download the script: update_retry_count_new.py.
2. Copy the script to your environment.
3. Connect to the HCX manager using the admin user.
4. Switch to the root user by running: su -.
5. Execute the script using the following command:
   • python update_retry_count.py 15
6. Repeat these steps on the peer HCX manager (Connector or Cloud, whichever applies).
7. Wait for 5 minutes before attempting the migration again.