Error - A generic error occurred in the vSphere Replication Management Server. Exception details: https:/xx.xxx.xxx.xx:8123/ invocation failed with "java.net.SocketTimeoutException: Read timed out".
search cancel

Error - A generic error occurred in the vSphere Replication Management Server. Exception details: https:/xx.xxx.xxx.xx:8123/ invocation failed with "java.net.SocketTimeoutException: Read timed out".

book

Article ID: 392023

calendar_today

Updated On: 03-26-2025

Products

VMware Live Recovery

Issue/Introduction

Symptoms: 

  • Registering Addon server with embedded shows below error.

    Error - A generic error occurred in the vSphere Replication Management Server. Exception details: https://xx.xxx.xxx.xx:8123/ invocation failed with "java.net.SocketTimeoutException: Read timed out".

  • Addon server shows disconnected.

Environment

VMware vSphere Replication 9.x
VMware vSphere Replication 8.x

Cause

  •  Embedded Replication server and Add-on Server are on different subnets, Since Embedded server needs to communicate with Add-on server over replication network we need to force this communication to go over the replication network.

  • Since there is no static routes configured on the appliances to reach the opposite site over the replication network the Addon fails to register.

  • From /opt/vmware/hms/logs/hms.log of Embedded Replication server 
    2025-03-25 14:50:38.429 INFO  com.vmware.hms [hms-main-thread-43] (..hms.util.HmsAsyncLock) [operationID=db1838d8-d667-4af8-b91c-e388617541f3-HMS-500,sessionID=DCAE1A84, task=HTID-5ee389ac-291b-4fde-b9b8-5e08a874c85d] | (Owner: 448:ServerRegistryHms.java:507:registerHbrServer,SiteManagerImpl.java:805:registerHbrServerWithTask(52fcdbe0-ca90-87d9-ada4-2343fa048dee), elapsed: 127821 milliseconds) releases the lock.
    2025-03-25 14:50:38.429 ERROR com.vmware.hms.remote.SiteManager [hms-main-thread-43] (..hms.remote.SiteManagerImpl) [operationID=db1838d8-d667-4af8-b91c-e388617541f3-HMS-500,sessionID=DCAE1A84, task=HTID-5ee389ac-291b-4fde-b9b8-5e08a874c85d] | Unable to register VR Server vm 'xxxx.xxx.xx.xx.xx' uri 'https://xx.xxx.xx.xx:8123' (ManagedObjectReference: type = VirtualMachine, value = vm-1595, serverGuid = a9929fb3-ec35-4ef0-a09d-1efa23b9041b)
    com.vmware.vim.vmomi.client.exception.ConnectionException: https://xx.xxx.xx.xx:8123/ invocation failed with "java.net.SocketTimeoutException: Read timed out"
          at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:265) ~[vlsi-client-9.0.2.jar:?]
          at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.setResponseError(HttpExchangeBase.java:362) ~[vlsi-client-9.0.2.jar:?]
          at com.vmware.vim.vmomi.client.http.impl.HttpExchange.invokeWithinScope(HttpExchange.java:59) ~[vlsi-client-9.0.2.jar:?]
          at com.vmware.vim.vmomi.core.tracing.NoopTracer$NoopSpan.runWithinSpanContext(NoopTracer.java:120) ~[vlsi-core-9.0.2.jar:?]
          at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:17) ~[vlsi-client-9.0.2.jar:?]
          at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.run(HttpExchangeBase.java:52) ~[vlsi-client-9.0.2.jar:?]
          at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:229) ~[vlsi-client-9.0.2.jar:?]
          at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:128) ~[vlsi-client-9.0.2.jar:?]
          at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:693) ~[vlsi-client-9.0.2.jar:?]
          at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:674) ~[vlsi-client-9.0.2.jar:?]
          at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:371) ~[vlsi-client-9.0.2.jar:?]
          at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:322) ~[vlsi-client-9.0.2.jar:?]
          at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:195) ~[vlsi-client-9.0.2.jar:?]
        at jdk.proxy2.$Proxy251.remove(Unknown Source) ~[?:?]
          at com.vmware.hms.net.ServerRegistryHms.prepareHbrServer(ServerRegistryHms.java:1235) ~[hms.jar:?]
          at com.vmware.hms.net.ServerRegistryHms.registerHbrServer(ServerRegistryHms.java:535) ~[hms.jar:?]
          at com.vmware.hms.remote.SiteManagerImpl.registerHbrServerWithTask(SiteManagerImpl.java:805) ~[hms.jar:?]
          at com.vmware.hms.remote.SiteManagerImpl$1.go(SiteManagerImpl.java:450) ~[hms.jar:?]
          at com.vmware.hms.task.TaskRunnable.run(TaskRunnable.java:86) ~[hms.jar:?]
          at com.vmware.hms.HmsTaskManager.lambda$0(HmsTaskManager.java:253) ~[hms.jar:?]
          at com.vmware.hms.util.executor.LoggerOpIdConfigurator$RunnableWithDiagnosticContext.run(LoggerOpIdConfigurator.java:132) ~[hms.jar:?]
          at com.vmware.hms.util.executor.LoggerOpIdConfigurator$2.run(LoggerOpIdConfigurator.java:99) ~[hms.jar:?]
          at com.vmware.jvsl.sessions.net.impl.TlsPreservingWrapper$2.run(TlsPreservingWrapper.java:47) ~[jvsl-sessions-9.0.2.jar:?]
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
          at java.lang.Thread.run(Unknown Source) [?:?]
          ... 23 more
    2025-03-25 14:50:38.429 DEBUG com.vmware.hms.phcollector.ops.VrOperationsSender [hms-main-thread-46] (..phcollector.ops.VrOperationsSender) [operationID=db1838d8-d667-4af8-b91c-e388617541f3-HMS-500,sessionID=DCAE1A84] | handleHmsTaskFinishedEvent[id: HTID-5ee389ac-291b-4fde-b9b8-5e08a874c85d; taskName: Register VR Server; taskManagedEntityId: null; taskTypeId: RegisterHbrTask; requestedExecutorId: hbr-management; taskTag: null; autoAbort: true; queueTime: 1742867309495; startTime: 1742867309507; completeTime: null; error:https://xx.xxx.xx.xx:8123/ invocation failed with "java.net.SocketTimeoutException: Read timed out"; success: false; result: null]
    2025-03-25 14:50:38.440 INFO  com.vmware.hms.net.hbr.svr.dc5cf904-28ef-4d49-ae94-d8ca397009cc.initialHbrConnection [hms-main-thread-43] (..impl.hbr.HbrServerGatewayImpl) [operationID=db1838d8-d667-4af8-b91c-e388617541f3-HMS-500,sessionID=DCAE1A84, task=HTID-5ee389ac-291b-4fde-b9b8-5e08a874c85d] | Stopping connection to VR Server xx.xxx.xx.xx:8123 ...
    2025-03-25 14:50:38.440 INFO  com.vmware.hms.net.hbr.ping.svr.dc5cf904-28ef-4d49-ae94-d8ca397009cc [hms-main-thread-43] (..net.impl.PersistentConnection) [operationID=db1838d8-d667-4af8-b91c-e388617541f3-HMS-500,sessionID=DCAE1A84, task=HTID-5ee389ac-291b-4fde-b9b85e08a874c85d] | Stopping connection to server xx.xxx.xx.xx:8123 for session key: site_903...3a79d4fb ...
    2025-03-25 14:50:38.440 INFO  com.vmware.hms.net.hbr.ping.svr.dc5cf904-28ef-4d49-ae94-d8ca397009cc [hms-main-thread-43] (..net.impl.PersistentConnection$ReconnectingPing) [operationID=db1838d8-d667-4af8-b91c-e388617541f3-HMS-500,sessionID=DCAE1A84, task=HTID-5ee389ac-291b-4fde-b9b8-5e08a874c85d] | Initiating shutdown of ping thread to server xx.xxx.xx.xx:8123for session key: site_903...3a79d4fb
  • The above error shows addon server unable to reach to embedded server.

  • From Add-on server drconfig.log shows below error:

Resolution

Perform below steps to add static route in vSphere replication server.

  • Check for interface configured in vSphere replication.
    Example : The first interface (eth0) has the default gateway, The second interface (eth1) is connected used for replication traffic. In some scenarios VRMS is configured with one interface (eth0) which serves both Management and replication traffic.

  • Replication traffic is supposed to go over (eth 1), Add static routes using the appliance SSH.
    Note: Based on Appliance network setup if only one interface configured then eth0, If two interface configured then eth1 to be used to add static route.

  • To add route run below command,
    route add -net Destination_Gateway/ Subnet_Mask Source_Gateway 
    Example : route add -net 19x.xxx.2xx.0/24 gw 19x.xxx.1xx.1

  • Route can also manually added by directly editing the interface file, open 10-ethx.network

    vi /etc/systemd/network/10-eth(x).network

    Add
    [Match]
    Name=eth1
    [Network]
    Address=<VRMS_Applaince_IP>/24
    DHCP=no
    Domains=XX.XXX.XX
    [DHCP]
    UseDNS=false
    
    [ROUTE]
    Destination=<Destination_Default_ Gateway>/<SubnetMask>
    Gateway=<Source_Default_Gateway>
  • Save the changes and Restart Network service by running below command,

    systemctl restart systemd-networkd

  • Register Add-on server with Embedded.