Enhanced replication stuck in connection state "Testing"

Products

• VMware Live Recovery

Issue/Introduction

Symptoms:

• When attempting to add a replication mapping, the Test process keeps running and does not complete.

  

• Unable to configure enhanced replication for virtual machines and it fails with the error:

  A replication error occurred at the vSphere Replication Server for replication 'vm_name'. Details: 'No connection to VR Server for virtual machine test on host source-esxihost in cluster management in cloud: Unknown
  
  

• The VLR UI displays below error,

  Unable to connect to HBR Management Server at https://###########:443/vrms. Reason: https://###########:443/vrms invocation failed with "java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-16071 [ACTIVE]" 30,000 milliseconds timeout on connection http-outgoing-16071 [ACTIVE]

Environment

VMware vSphere Replication 9.x
VMware Live Recovery 9.x

Cause

The issue occurs if there is network connectivity disruption between the HBR agent and the target system on port 32032. Due to this interruption, the HBR agent and HBR server lose communication, leading to heartbeat failures, host disconnect events, and missing broker mappings on the HBR server. These failures causes the services to enter a stale or inconsistent state, where existing sessions were no longer valid.

Although the network connectivity restored later(confirmed by successful ping and traceroute during troubleshooting), the HBR services did not automatically recover from the earlier failure. As a result, they continue to report connection issues until the services are manually restarted to re-establish proper communication.

From the logs of Source ESXI host /var/log/hbr-agent.log, indicates connection drop at target ESXi hosts.

2026-05-23T07:23:17.699Z In(166) hbr-agent-bin [4481030] : [0x0000009a8bbfc700] info: [Proxy [Group: GID-############ detected latency was 3498ps
2026-05-23T07:23:42.936Z In(166) hbr-agent-bin [4481030] : [0x0000009a8bafa700] error: [Proxy [Group: GID-############] -> [##.#.##.##:32032]] Failed to read from client: Connection reset by peer
2026-05-23T07:23:42.9362 In(166) hbr-agent-bin [4481030] : [0x0000009a8bb7b700] error: [Proxy [Group: GID-############] -> [##.#.##.##:32032] ] Failed to read from server: Operation canceled
2026-05-23T07:23:42.938Z In(166) hbr-agent-bin [4481030] : [0x0000009a8ba79700] info: [ConfigManager] No user configuration for key=hbrsvc target info in Configstore.
2026-05-23T07:23:42.9382 In(166) hbr-agent-bin [4481030] : [0x0000009a8ba79700] error: [ConfigManager] Failed to get config store object. Comp: esx, Grp: services, Key: hbrsv c_target_info, Id: ##.#.##.##, Prop: certificate
2026-05-23T07:23:42.9382 In (166) hbr-agent-bin [4481030] : [0x0000009a8ba79700] info: [ProxyConnection] Setting up secure tunnel to broker on ##.#.##.##:32032
2026-05-23T07:23:42.9382 In(166) hbr-agent-bin [4481030] :[0x0000009a8ba79700] info: (Proxy [Group: ] -> (##.#.##.##: 32032] ] Connecting to ##.#.##.##: 32032 without specific
2026-05-23T07:23:42.9382 In(166) hbr-agent-bin [4481030] :[0x0000009a8ba79700] info: [Proxy [Group: GID-############] -> [##.#.##.##:32032]] Setting
2026-05-23T07:23:42.9852 In(166) hbr-agent-bin [4481030] : [0x0000009a8bb7b700] info: [Proxy [Group: GID-############] -> [##.#.##.##:32032]] Connecting to ##.#.##.##:32032 without specific vmk
2026-05-23T07:23:42.9892 In(166) hbr-agent-bin [4481030] : [0x0000009a8bbfc700] info: [Proxy [Group: GID-############] -> [##.#.##.##:32032] ] TCP Connect latency was 3569ps
2026-05-23T07:23:43.013Z In(166) hbr-agent-bin [4481030] : [0x0000009a8bb7b700] error: [Proxy [Group: GID-############] -> [##.#.##.##:32032] ] Failed to login to brokered server: (8) The replication group was removed on the server side.
2026-05-23T07:23:43.0132 In(166) hbr-agent-bin [4481030] : [0x0000009a8bb7b700] error: [Proxy [Group: GID-############] -> [##.#.##.##:32032]] Failed to login to brokered server additional error info: The group was removed or failed-over.
2026-05-23T07:23:43.0132 In(166) hbr-agent-bin [4481030] : [0x0000009a8bb7b700] error: [Proxy [Group: GID-############] -> [##.#.##.##:32032] ] Exhausted all server endpoints reported by broker.
2026-05-23T07:23:43.3192 In(166) hbr-agent-bin [4481030] : [0x0000009a8bbfc700] error: [Proxy [Group: GID-############] -> [##.#.##.##:32032] ] Failed to read from client: Connection reset by peer
2026-05-23T07:23:43.319Z In(166) hbr-agent-bin [4481030] : [0x0000009a8bbfc700] error: [Proxy [Group: GID-############] -> [##.#.##.##:32032]] Failed to read from server: Operation canceled

 

The active replication connection to the target ESXi host drops unexpectedly (This is also occurring for target VLR/VRMS appliance). The "peer" (the target server) forcibly closed the connection. 

hms.log:

2026-07-14 20:35:11.803 ERROR com.vmware.hms.hbrsrvuw.healthmonitor.HealthChecksWorkflow [hms-main-thread-5] (..hbrsrvuw.healthmonitor.HealthChecksWorkflow) [] | Failed to setup ping test groups on peer site '37cb3dfe-2e06-443a-8b62-############' and replication mapping '(hms.ReplicationMapping) {
   dynamicType = null,
   dynamicProperty = null,
   localClusterMoId = domain-c100792,
   peerClusterMoId = domain-c59047
}'.
java.util.concurrent.ExecutionException: (hms.fault.LockingFault) {
   faultCause = null,
   faultMessage = null,
   originalMessage = Failed to write-lock object: PING-WORKFLOW-76550968-73da-422a-8bc7-############,
   lockToken = PING-WORKFLOW-76550968-73da-422a-8bc7-############
}
        at java.util.concurrent.CompletableFuture.reportGet(Unknown Source) ~[?:?]
        at java.util.concurrent.CompletableFuture.get(Unknown Source) ~[?:?]
        at com.vmware.hms.hbrsrvuw.healthmonitor.HealthChecksWorkflow.waitForPeerTask(HealthChecksWorkflow.java:280) ~[hms.jar:?]
        at com.vmware.hms.hbrsrvuw.healthmonitor.HealthChecksWorkflow.setupPingTestGroups(HealthChecksWorkflow.java:181) ~[hms.jar:?]
        at com.vmware.hms.hbrsrvuw.healthmonitor.HealthChecksWorkflow.executePingTest(HealthChecksWorkflow.java:158) ~[hms.jar:?]
        at com.vmware.hms.hbrsrvuw.healthmonitor.HealthChecksWorkflow.lambda$0(HealthChecksWorkflow.java:146) ~[hms.jar:?]
        at com.vmware.hms.hbrsrvuw.healthmonitor.HealthChecksExecutorService.doRun(HealthChecksExecutorService.java:82) ~[hms.jar:?]
        at com.vmware.hms.util.executor.LoggerOpIdConfigurator$RunnableWithDiagnosticContext.run(LoggerOpIdConfigurator.java:132) ~[hms.jar:?]
        at com.vmware.hms.util.executor.LoggerOpIdConfigurator$2.run(LoggerOpIdConfigurator.java:99) ~[hms.jar:?]
        at com.vmware.jvsl.sessions.net.impl.TlsPreservingWrapper$2.run(TlsPreservingWrapper.java:47) ~[jvsl-sessions-9.0.2.jar:?]
        at com.vmware.jvsl.executors.ThrottledExecutorService$1.run(ThrottledExecutorService.java:136) ~[jvsl-utils-9.0.2.jar:?]
        at com.vmware.hms.util.executor.LoggerOpIdConfigurator$RunnableWithDiagnosticContext.run(LoggerOpIdConfigurator.java:132) ~[hms.jar:?]
        at com.vmware.hms.util.executor.LoggerOpIdConfigurator$2.run(LoggerOpIdConfigurator.java:99) ~[hms.jar:?]
        at com.vmware.jvsl.sessions.net.impl.TlsPreservingWrapper$2.run(TlsPreservingWrapper.java:47) ~[jvsl-sessions-9.0.2.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: com.vmware.vim.binding.hms.fault.LockingFault: Object 'PING-WORKFLOW-76550968-73da-422a-8bc7-############' is locked by another ongoing operation in vSphere Replication Management Server. Try again later.

This indicates a VRMS deadlock, causing the ESXi host replication agents (HBR) to lose authentication and fail with continuous HTTP 500 errors and broken pipes. This proves that the ESXi hosts are reaching the replication appliances, but the management services on the appliances are crashing or failing to process the API requests due to the overarching service deadlock.

hbr-agent.log: 

2026-07-13T17:48:40.526Z In(166) hbr-agent-bin[17601332]: [0x0000000760e3c700] error: [Proxy [Group: GID-47475e60-07ea-35f0-9f35-2f76e6893f6c] -> [10.#.#.#:32032]] Failed to login to brokered server: (18) Login denied.
2026-07-13T17:48:52.070Z In(166) hbr-agent-bin[17601332]: [0x0000000760e3c700] error: [Proxy [Group: GID-d3e41ccc-35c4-4d62-b484-21bf1040a949] -> [10.#.#.#:32032]] Failed to login to brokered server: (18) Login denied. 

2026-07-10T18:29:26.067Z In(166) hbr-agent-bin[2099903]: [0x000000845ac48700] error: [Proxy [Group: GID-e49f17cb-bcbe-36dd-bab3-f699d2776597] -> [10.#.#.#:32032]] Failed to read from client: Connection reset by peer  
2026-07-10T18:29:26.067Z In(166) hbr-agent-bin[2099903]: [0x000000845ac48700] error: [Proxy [Group: GID-e49f17cb-bcbe-36dd-bab3-f699d2776597] -> [10.#.#.#:32032]] Failed to read from server: Operation canceled 

2026-07-10T18:30:08.390Z In(166) hbr-agent-bin[2099903]: [0x000000845ac48700] error: [Http] Unexpected HTTP status code: 500
2026-07-10T18:31:08.390Z In(166) hbr-agent-bin[2099903]: [0x000000845ab46700] error: [Http] Unexpected HTTP status code: 500
2026-07-10T18:32:08.389Z In(166) hbr-agent-bin[2099903]: [0x000000845ac48700] error: [Http] Unexpected HTTP status code: 500
2026-07-10T18:33:09.389Z In(166) hbr-agent-bin[2099903]: [0x000000845aac5700] error: [Http] Unexpected HTTP status code: 500
2026-07-10T18:34:09.389Z In(166) hbr-agent-bin[2099903]: [0x000000845ac48700] error: [Http] Unexpected HTTP status code: 500

2026-07-14T20:56:12.463Z Wa(180) vmkwarning: cpu57:16573490)WARNING: Hbr: 788: Failed to receive from 127.#.#.# (groupID=GID-a8acf9d6-0ae7-4ed2-a813-6da85e8f8cd0): Broken pipe
2026-07-14T20:56:12.463Z Wa(180) vmkwarning: cpu3:16986950)WARNING: Hbr: 788: Failed to receive from 127.#.#.# (groupID=GID-2386054a-27d5-3f59-9e00-2a433e896b2a): Broken pipe
2026-07-14T20:56:12.464Z Wa(180) vmkwarning: cpu25:17492629)WARNING: Hbr: 788: Failed to receive from 127.#.#.# (groupID=GID-ad659849-47e1-3d3e-abc9-796037d24131): Connection reset by peer

 

Resolution

1. Verify mappings under Enhanced Replication Mappings

2. If you are seeing deadlocks on the VRMS, reboot all the VRMS appliances including VR add-on servers. 

3. If you are seeing 'Login denied errors' on the ESXi hosts, refresh the site pairing and authentication tokens by reconnecting site pairs. 

4. If the VMs continue to show RPO violations or the hosts continue to log HTTP 500 errors 10-15 minutes after the VRMS services are restarted, the host agents may be stuck in a retry loop. Restart the hbr-agent & hbrsrv services on the host clusters used for replication. 

Commands to restart from CLI - 

/etc/init.d/hbr-agent restart

/etc/init.d/hbrsrv restart