HCX DR VM recovery stuck at 5%

Products

• VMware HCX

Issue/Introduction

• HCX DR VM recovery is stuck at 5%, and shows base sync is stuck at 0%

• Verified from HCX IX appliance logs (/var/log/vmware/hbrsrv.log).
  

  <time-stamp> java.lang.RuntimeException: ["Error connecting to HBR server <HBR-Server-IP>:8123. Reason: ConnectTimeoutException"]
  <time-stamp> [InterconnectService_SvcThread-58058, Ent: HybridityAdmin, , TxId: <Txn-ID>] ERROR c.v.v.h.a.hbr.HbrServerInstance- Error running request as communication to HBR server <HBR-Server-IP> failed

• ESXi host log (/var/run/log/vmkernel.log)- where source IX resides.
  

  <time-stamp> cpu1:6162855)WARNING: Hbr: 5093: Failed to establish connection to [<IX-IP>:31031 (groupID=HRID-<ID>): Timeout
  <time-stamp> cpu28:2103825)WARNING: Hbr: 5093: Failed to establish connection to [<IX-IP>:31031 (groupID=HRID-<ID>): Broken pipe

• RPO violation is observed in the HCX manager logs (/common/logs/admin/app.log).
  

  <time-stamp> [VsphereReplicationService_SvcThread-21153, T:'VRS Group Image Observer', R:<Job-ID>, S:COLLECT_REPLICATION_STAT, , TxId: <ID>] WARN  DR.INTERNAL.OBSERVER.VR- Another RPO voilation detected for replication : HRID-<HRID> voilations count : 4831

• ESXi host log (/var/run/log/vmkernel.log)- - where source guest VM resides.
  

  <time-stamp>WARNING: Hbr: 5093: Failed to establish connection to [<IX-IP>]:31031 (groupID=HRID-<ID>): Broken pipe
  <time-stamp>cpu0:5696779)WARNING: Hbr: 5093: Failed to establish connection to [<IX-IP>]:31031 (groupID=HRID-<ID>): Broken pipe
  <time-stamp> cpu8:6970351)WARNING: Hbr: 5093: Failed to establish connection to [<IX-IP>]:31031 (groupID=HRID-<ID>): Broken pipe

Environment

VMware HCX

Cause

Required ports are not whitelisted like TCP 31031, and other ports like 8123 are not whitelisted on the source side due to which RPO violation is encountered.

Resolution

Please ensure that the HCX Manager can communicate with the IX appliance on TCP ports 8123 and 31031. For details on all HCX communications and required ports, please visit: (VMware HCX ports and protocols)

Additional Information

Refer HBR RPO violations