HCX Migration Fails with “Initial Sync Failed – Error Running Request” even when concurrent migration limits is not exceeded.

Products

• VMware HCX

Issue/Introduction

HCX migration fails with “Initial Sync Failed – Error Running Request” even when concurrent migration limits have NOT been exceeded. For details on scenarios where concurrent migration limits are exceeded, please refer to KB: 428268

Environment

VMware HCX

Cause

Source HCX Manger logs:

/common/logs/admin/app.log:
YYYY-MM-DD HH:MM:SS.ms UTC [MobilityMigrationService_SvcThread-816, Ent: HybridityAdmin, , TxId: fec5a5ce-####-####-####-212ae449d764] INFO  c.v.h.s.m.jobs.MigrationTracker- Migration Starting -> Id: fc8d92c1-####-####-####-40218e014569::7bac2bbe-####-####-####-bba3f4d60ec5 ,  Type: vSphereReplication
YYYY-MM-DD HH:MM:SS.ms UTC [ReplicationTransferService_SvcThread-1900, Ent: HybridityAdmin, , TxId: fec5a5ce-####-####-####-212ae449d764] INFO  c.v.h.s.r.j.ReplicationTransferJob- Job (7fb6b617-####-####-####-ff3c3541caa4) Replication transfer job performing work in state:START_TRANSFER for transfer 7bac2bbe-####-####-####-bba3f4d60ec5
YYYY-MM-DD HH:MM:SS.ms UTC [ReplicationTransferService_SvcThread-1900, Ent: HybridityAdmin, , TxId: fec5a5ce-####-####-####-212ae449d764] INFO  c.v.v.h.m.utils.MobilityEventsUtil- [migId=7bac2bbe-####-####-####-bba3f4d60ec5] Service mesh for the migration is 'Service_Mesh_Name'
YYYY-MM-DD HH:MM:SS.ms UTC [ReplicationTransferService_SvcThread-1953, Ent: HybridityAdmin, , TxId: fec5a5ce-####-####-####-212ae449d764] INFO  c.v.h.s.r.j.ReplicationTransferJob- Job (7fb6b617-####-####-####-ff3c3541caa4) Replication transfer job performing work in state:SETUP_TARGET for transfer 7bac2bbe-####-####-####-bba3f4d60ec5
YYYY-MM-DD HH:MM:SS.ms UTC [ReplicationTransferService_SvcThread-2143, Ent: HybridityAdmin, , TxId: fec5a5ce-####-####-####-212ae449d764] INFO  c.v.h.s.r.j.ReplicationTransferJob- Job (7fb6b617-####-####-####-ff3c3541caa4) Replication transfer job performing work in state:SETUP_SOURCE for transfer 7bac2bbe-####-####-####-bba3f4d60ec5
YYYY-MM-DD HH:MM:SS.ms UTC [MobilityTransferService_SvcThread-590319, Ent: HybridityAdmin, , TxId: 7ed65325-####-####-####-ffac24c2c041] INFO  c.v.h.s.m.e.EndEstimatorJob- [MigrationEstimates] Job : e3a20803-ade6-4300-be37-531a3da13a0e , for transferId : 7bac2bbe-####-####-####-bba3f4d60ec5 current state is : ISSUE_END_COMMAND
YYYY-MM-DD HH:MM:SS.ms UTC [ReplicationTransferService_SvcThread-2133, Ent: HybridityAdmin, , TxId: fec5a5ce-####-####-####-212ae449d764] INFO  c.v.h.s.r.j.ReplicationTransferSourceCleanupJob- Job (72e718e5-####-####-####-999106f2ad4e)  Cleanup source job performing work in state:BEGIN for transfer 7bac2bbe-####-####-####-bba3f4d60ec5

Destination HCX manger logs:

/common/logs/admin/app.log:
YYYY-MM-DD HH:MM:SS.ms UTC [ReplicationTransferService_SvcThread-3673, Ent: HybridityAdmin, , TxId: TxId: cedb6645-####-####-####-7e2a06a1f6fd] ERROR c.v.v.h.a.hbr.HbrServerInstance- Error running request as communication to HBR server 10.##.##.103 failed
YYYY-MM-DD HH:MM:SS.ms UTC [ReplicationTransferService_SvcThread-3673, Ent: HybridityAdmin, , TxId: TxId: cedb6645-####-####-####-7e2a06a1f6fd] ERROR c.v.h.s.r.j.ContinuousReplicationJob- Error probing the target site for the group instance for the transfer 'd9856862-####-####-####-a2daae566d12'
com.vmware.vchs.hybridity.adapters.hbr.fault.HbrConnectionException: Error connecting to HBR server 10.##.##.103:8123. Reason: ConnectTimeoutException
        at com.vmware.vchs.hybridity.adapters.hbr.HbrConnection.executeRequest(HbrConnection.java:171)
        at com.vmware.vchs.hybridity.adapters.hbr.HbrSessionManager.loginBySSLThumbprint(HbrSessionManager.java:16)
        at com.vmware.vchs.hybridity.adapters.hbr.HbrServerInstance.login(HbrServerInstance.java:696)
        at com.vmware.vchs.hybridity.adapters.hbr.HbrAdapterFactory.setHbrInstance(HbrAdapterFactory.java:20)
        at com.vmware.vchs.hybridity.adapters.hbr.HbrAdapterFactory.getHbrServerInstance(HbrAdapterFactory.java:27)
        at com.vmware.vchs.hybridity.adapters.unified.hbr.UnifiedVrAdapterFactory.getHbrServerInstance(UnifiedVrAdapterFactory.java:62)
        at com.vmware.vchs.hybridity.adapters.unified.hbr.UnifiedVrAdapterFactory.setHbrInstance(UnifiedVrAdapterFactory.java:37)
        at com.vmware.vchs.hybridity.adapters.unified.hbr.UnifiedVrAdapterFactory.getUnifiedServerInstance(UnifiedVrAdapterFactory.java:50)
        at com.vmware.hybridity.service.replicationtransfer.utils.VrServerAdapterUtil.createVrServerAdapter(VrServerAdapterUtil.java:86)
        at com.vmware.hybridity.service.replicationtransfer.utils.VrServerAdapterUtil.getVrServerAdapter(VrServerAdapterUtil.java:76)
        at com.vmware.hybridity.service.replicationtransfer.jobs.ContinuousReplicationJob.doHaltReplication(ContinuousReplicationJob.java:685)
        at com.vmware.hybridity.service.replicationtransfer.jobs.ContinuousReplicationJob.run(ContinuousReplicationJob.java:105)
        at com.vmware.vchs.hybridity.messaging.LoggingJobWrapper.run(LoggingJobWrapper.java:41)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
YYYY-MM-DD HH:MM:SS.ms UTC [ReplicationTransferService_SvcThread-3667, Ent: HybridityAdmin, , TxId: TxId: 1e7f5d41-####-####-####-7e4f25d10788] INFO  c.v.h.s.r.utils.VrServerAdapterUtil- Creating new adapter instance for the HBR server '10.##.##.103'
YYYY-MM-DD HH:MM:SS.ms UTC [ReplicationTransferService_SvcThread-3673, Ent: HybridityAdmin, , TxId: TxId: 75eaef4b-####-####-####-f2e06837b57f] INFO  c.v.h.s.r.j.ContinuousReplicationJob- Job (8b177ca6-####-####-####-b8a44c526d0f) ContinuousReplicationJob performing work in state:HALT_REPLICATION for transfer 872d8a26-####-####-####-f5d025458cc5
YYYY-MM-DD HH:MM:SS.ms UTC [ReplicationTransferService_SvcThread-3667, Ent: HybridityAdmin, , TxId: TxId: 1e7f5d41-####-####-####-7e4f25d10788] ERROR c.v.h.s.r.j.ContinuousReplicationJob- Error probing the target site for the group instance for the transfer '02a58646-####-####-####-5873e0b71bdd'
java.lang.RuntimeException: Error Running request
        at com.vmware.vchs.hybridity.adapters.hbr.HbrServerInstance.login(HbrServerInstance.java:726)
        at com.vmware.vchs.hybridity.adapters.hbr.HbrAdapterFactory.setHbrInstance(HbrAdapterFactory.java:20)
        at com.vmware.vchs.hybridity.adapters.hbr.HbrAdapterFactory.getHbrServerInstance(HbrAdapterFactory.java:27)
        at com.vmware.vchs.hybridity.adapters.unified.hbr.UnifiedVrAdapterFactory.getHbrServerInstance(UnifiedVrAdapterFactory.java:62)
        at com.vmware.vchs.hybridity.adapters.unified.hbr.UnifiedVrAdapterFactory.setHbrInstance(UnifiedVrAdapterFactory.java:37)
        at com.vmware.vchs.hybridity.adapters.unified.hbr.UnifiedVrAdapterFactory.getUnifiedServerInstance(UnifiedVrAdapterFactory.java:50)
        at com.vmware.hybridity.service.replicationtransfer.utils.VrServerAdapterUtil.createVrServerAdapter(VrServerAdapterUtil.java:86)
        at com.vmware.hybridity.service.replicationtransfer.utils.VrServerAdapterUtil.getVrServerAdapter(VrServerAdapterUtil.java:76)
        at com.vmware.hybridity.service.replicationtransfer.jobs.ContinuousReplicationJob.doHaltReplication(ContinuousReplicationJob.java:685)
        at com.vmware.hybridity.service.replicationtransfer.jobs.ContinuousReplicationJob.run(ContinuousReplicationJob.java:105)
        at com.vmware.vchs.hybridity.messaging.LoggingJobWrapper.run(LoggingJobWrapper.java:41)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.vmware.vchs.hybridity.adapters.hbr.fault.HbrServiceNotReachableException: Error communicating to HBR server 10.##.##.103:8123. Reason: Service Unavailable
        at com.vmware.vchs.hybridity.adapters.hbr.HbrConnection.executeRequest(HbrConnection.java:233)
        at com.vmware.vchs.hybridity.adapters.hbr.HbrSessionManager.loginBySSLThumbprint(HbrSessionManager.java:16)
        at com.vmware.vchs.hybridity.adapters.hbr.HbrServerInstance.login(HbrServerInstance.java:696)
        ... 16 common frames omitted

If we check further this HBR server '10.##.##.103' corresponds to Service_Mesh_Name-IX-R1 (10.##.##.103 (Management and/or Replication))

Resolution

Please verify the following:

• Confirm that the service is running correctly on the IX appliance:
  • • Note: To log into an HCX Interconnect (IX) or Network Extension (NE) appliance via SSH, you must first connect to the main HCX Manager and then use its Central CLI (CCLI) to access the specific IX/NE appliance. You cannot SSH directly into the IX/NE appliance from an external client. For more information refer to the following document
      • Enabling SSH Access in HCX
      • How to SSH into the HCX Interconnect (IX) or Network Extension (NE) appliance
  • systemctl status hbrsrv
• If the service is not running as expected, restart the Host-Based Replication service:
  • systemctl restart hbrsrv
• Re-verify the service status:
  • systemctl status hbrsrv
• Check connectivity between the Destination HCX Manager and the Destination IX appliance (Service_Mesh_Name-IX-R1 – 10.##.##.103) over port 8123 (Management and/or Replication network).