HCX Bulk Migration fails during Switchover with "FileNotFound hbrimagedisk.vmdk" error

Products

• VMware HCX

Issue/Introduction

VMware HCX Bulk Migrations fail during the switchover or power-on phase at the target site. The target vCenter Server reports a disk error preventing the Virtual Machine from starting.

Symptoms:

• The migration task fails specifically during the "vpx.vmprov.PowerOnVm" action.

• Target vCenter vpxd.log contains the following error: 

  error vpxd[<ID>] [Originator@6876 sub=VmProv opID=<ID>] Get exception while executing action vpx.vmprov.PowerOnVm: (vim.fault.FileNotFound) { message = "VMware ESX cannot find the virtual disk "/vmfs/volumes/<DATASTORE_ID>/<VM_ID>/hbrimagedisk.RDID-<ID>.vmdk". Verify the path is valid and try again." }

Environment

VMware HCX 4.11.#

Cause

A transient connectivity issue between the target HCX Interconnect (IX) appliance and the ESX host causes the IX HBRSRV service to lose track of a successful disk consolidation task.

While the consolidation task completes on the ESX host (removing the hbrimagedisk), the IX appliance, unaware of the completion, provides the now-obsolete hbrimagedisk specification to the target vCenter during the VM instantiation phase.

Resolution

The current workaround for this issue is to retry the RAV migration.

If you believe you have encountered this issue, please open a support case with Broadcom and provide the below information.

• Migration ID and VM name which failed switchover.
• Source/Target HCX Manager log bundle with IX and DB dump selected.
• ALL ESXi host log bundles from TARGET site.
• Perform the following openssl command from HCX IX-R appliance to each ESXi host in Target cluster
  
  • openssl s_client -connect ESXi_IP:443
    
    • For steps on how to connect to the IX appliance, please see: Using Central CLI to Connect to HCX Services

Additional Information

Subscribe to this knowledge article to get updates on this issue.