Error: "Unable to retrieve pairs from extension server at https://<vr-server>:8043.Unable to login to 'HBR Management Server at https://<VR-Server>:8043'."

Products

VMware Live Recovery

Issue/Introduction

Symptoms:

After upgrading replication appliance on production from 9.0.0. to 9.0.2 Replication appliance is Unable to login to 'HBR Management Server from one site to the other site.
On the SRM UI page you will see this error message as shown below: Unable to retrieve pairs from extension server at https://v##-##########100.##.h####.###.##.##:8043.Unable to login to 'HBR Management Server at https://v##-r#######0.##.h####.##.##.##:8043'
Operation ID: 4d6f39b3-############-51625#####3b
vSphere Replication appliances on SRM UI page shows as disconnected and unknown.

Environment

VMware vSphere Replication 9.x

Cause

Replication appliance has difficulty in resolving the FQDN names due to DNS issues.

Below details found from /opt/vmware/hms/logs/hms.log,

2025-04-16 05:06:15.359 DEBUG com.vmware.hms.host.HbrServerSourceHostSyncRequest [hms-main-scheduled-thread-1] (..hms.host.SourceHostSyncRequest) [] | SetAllHostsInHbrServer[hbrServerId: 5220f622-6515-ac45-19f0-824e915132b4] process ...
2025-04-16 05:06:15.360 ERROR com.vmware.hms.host.HbrServerSourceHostSyncTask [hms-main-scheduled-thread-1] (..hms.host.SourceHostSyncTask$RequestProcessor) [] | Error executing SetAllHostsInHbrServer[hbrServerId: 5220f622-6515-ac45-19f0-824e915132b4]
java.lang.IllegalStateException: Connection context is not set for server view HbrServerViewImpl
        at com.vmware.jvsl.sessions.net.ServerView.createStub(ServerView.java:42) ~[jvsl-sessions-9.0.2.jar:?]
        at com.vmware.hms.net.impl.hbr.HbrServerViewImpl.getReplicationManager(HbrServerViewImpl.java:60) ~[hms.jar:?]
        at com.vmware.hms.host.HbrServerSourceHostSyncRequest.getTarget(HbrServerSourceHostSyncRequest.java:38) ~[hms.jar:?]
        at com.vmware.hms.host.HbrServerSourceHostSyncRequest.getTarget(HbrServerSourceHostSyncRequest.java:1) ~[hms.jar:?]
        at com.vmware.hms.host.SourceHostSyncRequest.run(SourceHostSyncRequest.java:39) ~[hms.jar:?]
        at com.vmware.hms.host.SourceHostSyncTask$RequestProcessor.run(SourceHostSyncTask.java:57) ~[hms.jar:?]
        at com.vmware.hms.util.executor.LoggerOpIdConfigurator$RunnableWithDiagnosticContext.run(LoggerOpIdConfigurator.java:132) ~[hms.jar:?]
        at com.vmware.hms.util.executor.LoggerOpIdConfigurator$2.run(LoggerOpIdConfigurator.java:99) ~[hms.jar:?]
        at com.vmware.jvsl.sessions.net.impl.TlsPreservingWrapper$2.run(TlsPreservingWrapper.java:47) ~[jvsl-sessions-9.0.2.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]

The connection error shows host sync issues on hbrserver at target side.

Also From /opt/vmware/hms/logs/hms.log found,

2025-05-29 09:08:10.589 WARN com.vmware.jvsl.util.RequestContextProviderImpl [hbr-metrics-thread-2] (..jvsl.util.RequestContextProviderImpl) [] | Generating operation id for request (MoRef: 'MoRef: type = HbrServerManager, value = HbrServerManager, serverGuid = null', method: 'getMetricsAndEvents', endpointUri: 'https://##.###.##.###:443/hbr', operationID: '3af5459b-02da-4fa5-b27c-ed44c2066104')
2025-05-29 09:08:10.781 ERROR com.vmware.hms.net.hbr.ping.svr.1f8e874c-5675-11ec-8875-0a94eff803dd [hms-ping-scheduled-thread-7] (..net.impl.VmomiPingConnectionHandler) [operationID=7a1bb6b8-4898-443a-b5bb-e819f58c7be4-HMS-PING] | Ping for server ##.###.##.###:443/hbr for session: N/A failed:
com.vmware.vim.binding.vmodl.fault.SystemError: null
at jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(Unknown Source) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Unknown Source) ~[?:?]
at java.lang.reflect.ReflectAccess.newInstance(Unknown Source) ~[?:?]
at jdk.internal.reflect.ReflectionFactory.newInstance(Unknown Source) ~[?:?]
at java.lang.Class.newInstance(Unknown Source) ~[?:?]

Ping to Hosts fails, this occurs if DNS fails to resolve.

Resolution

Fix the DNS issue, resolving the name resolution will effectively address the problem.

Workaround:

Take snapshot of the VRMS appliances before making any changes on the VRMS appliance.

To work around this issue, add the vCenter, SRM and VRMS IP and FQDN for both paired sites in the /etc/hosts file on both SRM, vCenter and VR appliances.
To add the vCenter IP and FQDN:

Log in to vSphere Replication manager console.
Run these commands:

cd /etc
cp hosts hosts.bak (to create back for host file)
vi hosts
Enter I key to enable insert mode.
Move the cursor, create an entry for vCenter, SRM and VRMS IP and FQDN for both sites below the local host entry.

Example of host's file:

# VAMI_EDIT_BEGIN
127.#.#.# localhost
192.###.#.## vr.example.local vr
# VAMI_EDIT_END
Add the vCenter IP and FQDN after vSphere Replication IP and FQDN.

Example of hosts file after the change:

# VAMI_EDIT_BEGIN
127.#.#.# localhost
192.168.#.## srmprod.example.local srm
192.167.#.## vcprod.example.local vcprod
192.166.#.## vrprod.example.local vr
192.168.#.## srmdr.example.local srm
192.167.#.## vcproddr.example.local vcprod
192.166.#.## vrdr.example.local vr
# VAMI_EDIT_END
Refresh the SRM DR UI page/Reconnect the Pair.

After adding and refreshing the SRM GUI page, validate for issue resolution.

Additional Information

Editing files on an ESXi host using vi editor