Attachment of vLCR recovery SDDC will fail with connection time out to NSX Manager.
search cancel

Attachment of vLCR recovery SDDC will fail with connection time out to NSX Manager.

book

Article ID: 313733

calendar_today

Updated On:

Products

VMware Live Recovery VMware Cloud on AWS

Issue/Introduction

 

  • Attaching recovery SDDC in VLCR UI may fail.
  • irrAgent logs show below events 


2023-08-28T02:02:08.639+0000 [.qtp1686017373-1288765] Caught exception while configureSddc: com.vmware.vapi.client.exception.ConnectionException: Connection timed
 out (Read failed)
        at com.vmware.vapi.internal.protocol.client.rpc.http.ApacheClientRestTransport.execute(ApacheClientRestTransport.java:81)
        at com.vmware.vapi.internal.protocol.client.rest.DefaultRequestExecutorFactory$DefaultRequestExecutor.execute(DefaultRequestExecutorFactory.java:45)
        at com.vmware.vapi.internal.protocol.client.rest.RestClientApiProvider.invoke(RestClientApiProvider.java:67)
        at com.vmware.vapi.internal.bindings.Stub.invoke(Stub.java:241)
        at com.vmware.vapi.internal.bindings.Stub.invokeMethodAsync(Stub.java:191)
        at com.vmware.vapi.internal.bindings.Stub.invokeMethod(Stub.java:137)
        at com.vmware.nsx_policy.infra.domains.gateway_policies.RulesStub.patch(RulesStub.java:189)
        at com.vmware.nsx_policy.infra.domains.gateway_policies.RulesStub.patch(RulesStub.java:177)
        at com.datrium.vmcdr.vsphere.VmcClient.createComputeFirewalls(VmcClient.java:1207)
        at com.datrium.vmcdr.vsphere.VmcClient.configureSddc(VmcClient.java:1692)
        at com.datrium.vmcdr.vsphere.VmcClient.configureSddc(VmcClient.java:2274)
        at com.datrium.irr.api.DevVmcCommand.configure_sddc(DevVmcCommand.java:882)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

  • API calls from VLCR to NSX manager fails 

    Exception: Connect to nsxManager.sddc-18-178-235-110.vmwarevmc.com:443 [nsxManager.sddc-xx-xxx-xxx-xxx.xxxxxxx.xxx/xx.xxx.xx.xxx] failed: connect timed out"

Environment

VMware Live Cyber Recovery 7.26.x

VMware Cloud on AWS

Cause

VLCR needs NSX Manager connectivity to automatically program the firewall rules required for communication between different VLCR entities.

A CSP authentication using a refresh token that has many roles assigned to it, results in a larger header for every REST API call to NSX-Manager. This causes the connection to hang with just a few number to API calls. 

When refresh token with a minimal set of roles assigned to it is used, the efficiency of number REST API calls over the TCP connection increases and does not saturate connection.

Resolution

VMware by Broadcom is aware of this issue and is currently working on a permanent fix. 


Workaround:

  1. Generate a new API token with minimum roles required for VLCR. The following documentation can be referred for the same.
  2. Once the new token generated, update the same in VLCR.
  3. After replacing the token open a support case with VMware by Broadcom to restart the irrAgent on the CDVX as the older token might be cached in it.
  4. Try re-attaching the SDDC in VLCR UI. 


Note: API Token is a legacy process for authentication now. VMware Live Cyber Recovery leverages OAuth 2.0 to communicate with VMware Cloud Services backend services and VMware Cloud on AWS including NSX Manager. Refer Authorize access for VLCR