During a HCX Bulk Migration, 3rd party Automation Application reports failure but VMs are powered on at the destination (VM Duplication)
Article ID: 448705
Updated On:
Products
Issue/Introduction
During an VMware HCX Bulk Migration, the 3rd party application may report a "failure" status during the switchover phase. Despite the reported failure, the Virtual Machines (VMs) successfully register and power on at the destination site. Crucially, the source VMs may also be found in a powered-on state, leading to a "split-brain" condition or VM duplication across vCenters.
Observed Symptoms:
- HCX Manager reports: the source Virtual machine is powered off.
- Destination vCenter shows the VM as registered and Powered On.
- It is observed a short ime later after the migration the source vCenter showing the VM as Powered On despite HCX orchestration powering off the VM.
Environment
VMware HCX
Third-party automation (e.g., Ansible, Terraform) managing VM power states.
Cause
In verified cases, a service account (such as an Ansible template) was configured with a "fail-safe" reboot or power-on logic. If the HCX switchover process exceeds a specific internal timer in the automation tool (e.g., a 90-minute wait period), the automation tool prematurely issues a PowerOn command to the source VM.
The State of a VM and/or cuase of a VM event can be viewed in the vCenter by highlighting the VM -> Choosing the Monitor tab -> and reviewing the Tasks and Events section in the vSphere client
Resolution
To resolve this issue, the external automation must be aligned with the HCX migration/switchover windows:
- Automation Coordination:
- Identify any service accounts or automation templates (Ansible, etc.) that manage VM power states.
- Disable or extend "fail-safe" power-on timers during scheduled HCX migration windows to ensure they do not conflict with the switchover process.
- Manual Reconciliation:
- In the event of a duplication, manually verify which instance (Source or Destination) contains the most current data.
- Gracefully shut down and decommission the stale source VM instance to resolve the split-brain condition.
Additional Information
Feedback
