[VMConAWS] [VCDR] Attachment of VCDR recovery SDDC will fail with connection time out to NSX Manager.

Products

• VMware Live Recovery
• VMware Cloud on AWS

Issue/Introduction

Issue identification and workaround to remediate the issue

Symptoms:
Attaching recovery SDDC in VCDR UI may fail. 

irrAgent logs show below events 

2023-08-28T02:02:08.639+0000 [.qtp1686017373-1288765] Caught exception while configureSddc: com.vmware.vapi.client.exception.ConnectionException: Connection timed
 out (Read failed)
        at com.vmware.vapi.internal.protocol.client.rpc.http.ApacheClientRestTransport.execute(ApacheClientRestTransport.java:81)
        at com.vmware.vapi.internal.protocol.client.rest.DefaultRequestExecutorFactory$DefaultRequestExecutor.execute(DefaultRequestExecutorFactory.java:45)
        at com.vmware.vapi.internal.protocol.client.rest.RestClientApiProvider.invoke(RestClientApiProvider.java:67)
        at com.vmware.vapi.internal.bindings.Stub.invoke(Stub.java:241)
        at com.vmware.vapi.internal.bindings.Stub.invokeMethodAsync(Stub.java:191)
        at com.vmware.vapi.internal.bindings.Stub.invokeMethod(Stub.java:137)
        at com.vmware.nsx_policy.infra.domains.gateway_policies.RulesStub.patch(RulesStub.java:189)
        at com.vmware.nsx_policy.infra.domains.gateway_policies.RulesStub.patch(RulesStub.java:177)
        at com.datrium.vmcdr.vsphere.VmcClient.createComputeFirewalls(VmcClient.java:1207)
        at com.datrium.vmcdr.vsphere.VmcClient.configureSddc(VmcClient.java:1692)
        at com.datrium.vmcdr.vsphere.VmcClient.configureSddc(VmcClient.java:2274)
        at com.datrium.irr.api.DevVmcCommand.configure_sddc(DevVmcCommand.java:882)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

API calls from VCDR to NSX manager fails 

Exception: Connect to nsxManager.sddc-18-178-235-110.vmwarevmc.com:443 [nsxManager.sddc-xx-xxx-xxx-xxx.xxxxxxx.xxx/xx.xxx.xx.xxx] failed: connect timed out"

Cause

VCDR needs NSX Manager connectivity to automatically program the firewall rules required for communication between different VCDR entities.
A CSP authentication using a refresh token that has many roles assigned to it, results in a larger header for every REST API call to NSX-Manager. This causes the connection to hang with just a few number to API calls. 

When refresh token with a minimal set of roles assigned to it is used, the efficiency of number REST API calls over the TCP connection increases and does not saturate connection.

Resolution

VMware is aware of this issue and is currently working on a permanent fix. 

Workaround:

1. Generate a new API token with minimum roles required for VCDR. The following documentation can be referred for the same.
2. Once the new token generated, update the same in VCDR as shown below.
3. After replacing the token engage VMware Support to restart the irrAgent on the CDVX as the older token might be cached in it.
4. If the SDDC which failed to attach before is being attached again then the network segment named “sddc-cloud-dr-proxy-network” has to be deleted manually first before attempting to re-attach. This manual deletion can be done from the Network and Security tab of the SDDC accessed via the VMC console as seen below.
5. Try re-attaching the SDDC in VCDR UI.