Attachment of vLCR recovery SDDC will fail with connection time out to NSX Manager.
search cancel

Attachment of vLCR recovery SDDC will fail with connection time out to NSX Manager.

book

Article ID: 313733

calendar_today

Updated On: 01-16-2025

Products

VMware Live Recovery VMware Cloud on AWS

Issue/Introduction

Symptoms:

  • Attaching recovery SDDC in vLCR UI may fail.
  • irrAgent logs show below events 


2023-08-28T02:02:08.639+0000 [.qtp1686017373-1288765] Caught exception while configureSddc: com.vmware.vapi.client.exception.ConnectionException: Connection timed
 out (Read failed)
        at com.vmware.vapi.internal.protocol.client.rpc.http.ApacheClientRestTransport.execute(ApacheClientRestTransport.java:81)
        at com.vmware.vapi.internal.protocol.client.rest.DefaultRequestExecutorFactory$DefaultRequestExecutor.execute(DefaultRequestExecutorFactory.java:45)
        at com.vmware.vapi.internal.protocol.client.rest.RestClientApiProvider.invoke(RestClientApiProvider.java:67)
        at com.vmware.vapi.internal.bindings.Stub.invoke(Stub.java:241)
        at com.vmware.vapi.internal.bindings.Stub.invokeMethodAsync(Stub.java:191)
        at com.vmware.vapi.internal.bindings.Stub.invokeMethod(Stub.java:137)
        at com.vmware.nsx_policy.infra.domains.gateway_policies.RulesStub.patch(RulesStub.java:189)
        at com.vmware.nsx_policy.infra.domains.gateway_policies.RulesStub.patch(RulesStub.java:177)
        at com.datrium.vmcdr.vsphere.VmcClient.createComputeFirewalls(VmcClient.java:1207)
        at com.datrium.vmcdr.vsphere.VmcClient.configureSddc(VmcClient.java:1692)
        at com.datrium.vmcdr.vsphere.VmcClient.configureSddc(VmcClient.java:2274)
        at com.datrium.irr.api.DevVmcCommand.configure_sddc(DevVmcCommand.java:882)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

  • API calls from vLCR to NSX manager fails 

    Exception: Connect to nsxManager.sddc-18-178-235-110.vmwarevmc.com:443 [nsxManager.sddc-xx-xxx-xxx-xxx.xxxxxxx.xxx/xx.xxx.xx.xxx] failed: connect timed out"

Environment

VMware Live Cyber Recovery

VMware Cloud on AWS

Cause

vLCR needs NSX Manager connectivity to automatically program the firewall rules required for communication between different VCDR entities.
A CSP authentication using a refresh token that has many roles assigned to it, results in a larger header for every REST API call to NSX-Manager. This causes the connection to hang with just a few number to API calls. 

When refresh token with a minimal set of roles assigned to it is used, the efficiency of number REST API calls over the TCP connection increases and does not saturate connection.

Resolution

Broadcom is aware of this issue and is currently working on a permanent fix. 



Workaround:

  1. Generate a new API token with minimum roles required for VCDR. The following documentation can be referred for the same.
  2. Once the new token generated, update the same in VCDR as shown below.
  3. After replacing the token engage VMware Support to restart the irrAgent on the CDVX as the older token might be cached in it.
  4. If the SDDC which failed to attach before is being attached again then the network segment named “sddc-cloud-dr-proxy-network” has to be deleted manually first before attempting to re-attach. This manual deletion can be done from the Network and Security tab of the SDDC accessed via the VMC console as seen below.
  5. Try re-attaching the SDDC in VCDR UI.