HCX - Migration is not progressing with the error: "Error Running request"

Products

• VMware HCX

Issue/Introduction

• HCX migrations are stuck at 0% during the 'Transfer Started' phase OR
• HCX Bulk Migration is failing for VMs during the "Setting disk UUIDs on target" stage.
• Bulk migration may fail with the error: "Error Running request" during the "Setting disk UUIDs on target" stage.
  
  
  
• The following errors are observed in /common/logs/admin/app.log from HCX Manager source OR target:
  

  <timestamps> [InterconnectService_SvcThread-4375, Ent: HybridityAdmin, , TxId: ########-####-####-####-############] ERROR c.v.v.h.s.i.ThumbprintExchanger- Error registering host 'host-###' with the HBR appliance '########-####-####-####-############'java.lang.RuntimeException: Error Running request
  
  or
  
  <timestamps> [InterconnectService_SvcThread-2861, Ent: HybridityAdmin, , TxId: ########-####-####-####-############] ERROR c.v.v.h.a.hbr.HbrServerInstance- Error running request as communication to HBR server <IX-IP> failed
  
  java.lang.RuntimeException: ["Error connecting to HBR server <IX-IP>:8123. Reason: ConnectTimeoutException"]

• The Base sync may be stuck at 0%:
  

  <timestamps> [ReplicationTransferService_SvcThread-603, Ent: HybridityAdmin, , TxId:########-####-####-####-############] INFO c.v.h.s.r.j.ReplicationTransferMonitor- Source side replication status is: {"state":"syncing","progressInfo" {"progress":0,"bytes_transferred":0,"bytes_to_transfer":0,"checksum_total_bytes":419628580864,"checksum_compared_bytes":0}} for vm: ##### for transfer########-####-####-####-############
  
  <timestamps> [InterconnectService_SvcThread-3625, Ent: HybridityAdmin, , TxId:########-####-####-####-############] ERROR c.v.v.h.s.i.ThumbprintExchanger- Error registering host 'host-###' with the HBR appliance '########-####-####-####-############

Cause

The potential cause for this failure is related to EMPTY ESXi host information found to be on HBR server running on a given IX appliance at source/target side.
In general, HCX manager populates the ESXi host information to each IX appliances running on local environment through an HTTPS call to the management IP corresponding to those IX appliances/HBR server.

When a user configures a proxy server, it is required to add an exception for IX appliance management IP addresses via HCX appliance-management 9443 UI to have a successful HTTPS call between HCX manager and the HBR server, otherwise HBR server will not able to get any host information from HCX manager to be used by vSR replication workflow.

Refer to the Configure a Proxy Server

Resolution

Please follow below steps to remediate the HTTPS connection issues between HCX manager and IX/HBR server:

Step:1 Add an exception for IX appliance management IP addresses via HCX appliance-management 9443 UI as shown below:
GO to HCX 9443 UI >> Administration >> Proxy >> Proxy Server Configuration >> Edit >> Add IX-Appliance Mgmt IP under Proxy Exclusions >> Save

Step:2  Restart App-engine service from HCX 9443 UI

If you are not using a proxy, ensure that the HCX Manager can communicate with the IX appliance on port 8123.
For details on all HCX communications and required ports, please visit: VMware Ports and Protocols

Additional Information

This will ONLY impact vSR Bulk and RAV Migration which uses HBR functionality.
vMotion & Cold Migration will remain functional.
Network Extension services will remain unaffected.