HCX Migration fails at switchover with error "Failed to instantiate vm. Task state: error"

Products

• VMware HCX

Issue/Introduction

• Bulk migration syncs the VM data to remote location and switch over fails with the following error: "Failed to instantiate vm. Task state: error"
  
  
  
  
  
• /common/logs/admin/app.log shows the following error:
  
  

  <timestamps> UTC [ReplicationTransferService_SvcThread-572, Ent: DEFAULT, , TxId: ####-####-f38ea19ad178] ERROR c.v.v.h.a.vcenter.VCenterAdapter- Instantiating a VM <Virtual-Machine-name> failed.
  com.vmware.vim.binding.vim.fault.DatacenterMismatch: The input arguments had entities that did not belong to the same datacenter.
  
  <timestamps> UTC [ReplicationTransferService_SvcThread-572, Ent: DEFAULT, , TxId: ####-####-f38ea19ad178] ERROR c.v.h.s.r.jobs.InstantiateVmJob- Error Executing Job (####-####-27c7d594df87)
  java.lang.RuntimeException: Failed to instantiate vm. Task state: error
  Caused by: com.vmware.vim.binding.vim.fault.DatacenterMismatch: The input arguments had entities that did not belong to the same datacenter.
  at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

   

• /var/log/vmware/vpxd/vpxd.log shows datacenter mismatch :
  
  

  <timestamps> info vpxd[30015] [Originator@6876 sub=vpxLro opID=####-####-f38ea19ad178-bc] [VpxLRO] -- BEGIN task-11458751 -- group-v#### -- vim.Folder.registerVm -- ####-####-32d3fc4c4c06(####-####-cb2805e47f58)
  <timestamps> error vpxd[30015] [Originator@6876 sub=vmProvCheckUtil opID=####-####-f38ea19ad178-bc] Entity [vim.Folder:group-v####,Discovered virtual machine] not in datacenter [Datacenter-Name]
  <timestamps> warning vpxd[30015] [Originator@6876 sub=Vmomi opID=####-####-f38ea19ad178-bc] VMOMI activation LRO failed; <<####-####-32d3fc4c4c06, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 34156'>>, group-v####, vim.Folder.registerVm, <vim.version.version10, official, 6.0>,
   (null)>, N3Vim5Fault18DatacenterMismatch9ExceptionE(Fault cause: vim.fault.DatacenterMismatch
  
  <timestamps> error vpxd[30015] [Originator@6876 sub=Default opID=####-####-f38ea19ad178-bc] [VpxLRO] -- ERROR task-11458751 -- ####-####-32d3fc4c4c06(####-####-cb2805e47f58) -- group-v#### -- vim.Folder.registerVm: :vim.fault.DatacenterMismatch
  --> Result:
  --> (vim.fault.DatacenterMismatch) {
  -->    faultCause = (vmodl.MethodFault) null,
  -->    faultMessage = <unset>,
  -->    invalidArgument = (vim.fault.DatacenterMismatch.Argument) [
  -->       (vim.fault.DatacenterMismatch.Argument) {
  -->          entity = 'vim.Folder:####-####-1fa57fa25d22:group-v####',
  -->          inputDatacenter = 'vim.Datacenter:####-####-1fa57fa25d22:datacenter-#6'
  -->       }
  -->    ],
  -->    expectedDatacenter = 'vim.Datacenter:####-####-1fa57fa25d22:datacenter-#1'
  -->    msg = ""

  
  
  
  

 

Environment

VMware HCX

Cause

The vpxd logs indicate an InvalidArgument fault caused by a Datacenter Mismatch. Specifically, the object group-v### is located in a different datacenter than the one expected by the current process. This typically occurs when a VM, folder, or network is being moved or mapped to a destination that does not align with its source datacenter's administrative boundary.

Resolution

To resolve this issue, resync vCenter to HCX to fix the inventory mismatch.

• Login to the HCX Appliance Management (VAMI) on port 9443 with "admin" credentials (Example: https://<hcx-connector-fqdn or ip address>:9443)
• On the "Dashboard", find the "vCenter" tile and click "Manage"
• On the "vCenter Server" tile click "Edit"
• Input the username and password for the vCenter Server service account (Example: [email protected]) and click "Save"
• Go to the "Appliance Summary" tab and stop/start app-engine and web-engine services
• Retry your migration