IP Customization fails - A general system error occurred: N7Vmacore15SystemExceptionE Connection timed out

Products

• VMware Live Recovery

Issue/Introduction

1. VMs get stuck on the 'Customize IP' step when running a recovery plan 

2. TEST Recovery process will take a long time before it fails 

3. PLANNED MIGRATION can also fail with the same errors 

Error - A general system error occurred: N7Vmacore15SystemExceptionE Connection timed out (110) [context]zKq7AVECAAQAAAFuSQELdm13YXJlLWRyAADtoRxsaWJ2bWFjb3JlLnNvAAC00zAAJUw2AOZMNgDVUzYAzFk2AE0BMQCtGDEADiVBAYd/AGxpYnB0aHJlYWQuc28uMAACLzYPbGliYy5zby42AA==[/context] [backtrace begin] product: VMware vCenter Site Recovery Manager, version: 8.7.0, build: build-21589505, tag: vmware-dr, cpu: x86_64, os: linux, 
buildType: release 
backtrace[03] libvmacore.so[0x001CA1ED] 
backtrace[04] libvmacore.so[0x0030D3B4] 
backtrace[05] libvmacore.so[0x00364C25] 
backtrace[06] libvmacore.so[0x00364CE6] 
backtrace[07] libvmacore.so[0x003653D5] 
backtrace[08] libvmacore.so[0x003659CC] 
backtrace[09] libvmacore.so[0x0031014D] 
backtrace[10] libvmacore.so[0x003118AD] 
backtrace[11] libvmacore.so[0x0041250E] 
backtrace[12] libpthread.so.0[0x00007F87] 
backtrace[13] libc.so.6[0x000F362F] 
[backtrace end]

VMware-dr.log:

2024-10-30T18:16:52.559Z error vmware-dr[29747] [SRM@6876 sub=Recovery ctxID=39a014ae opID=6e60fed3-98a4-41f0-9d25-1581993ba196-test:b14e:1f0e:a48b] [cbf92c2d-a335-4886-b2ad-d4134fcbe8f6.failoverOrchJob] IP customization failed for VM Broadcom [vm-101128]: (dr.fault.Cancelled) {
 
2024-10-30T17:27:55.831Z warning vmware-dr[29744] [SRM@6876 sub=HttpConnectionPool-000000 opID=e92babb6-89fe-4235-bcb9-63cf21641aa5-test:fb7b:4589:c812:2f0d:650d] Failed to get pooled connection; <cs p:00007fb686009260, TCP:VMware.support.local:443>, (null), duration: 130315msec, N7Vmacore15SystemExceptionE(Connection timed out)
--> [context]zKq7AVECAAQAAAFuSQELdm13YXJlLWRyAADtoRxsaWJ2bWFjb3JlLnNvAAC00zAAJUw2AOZMNgDVUzYAzFk2AE0BMQCtGDEADiVBAYd/AGxpYnB0aHJlYWQuc28uMAACLzYPbGliYy5zby42AA==[/context]
2024-10-30T17:27:55.832Z info vmware-dr[29744] [SRM@6876 sub=IO.Http opID=e92babb6-89fe-4235-bcb9-63cf21641aa5-test:fb7b:4589:c812:2f0d:650d] Set user agent error; state: 1, (null), N7Vmacore15SystemExceptionE(Connection timed out)
--> [context]zKq7AVECAAQAAAFuSQELdm13YXJlLWRyAADtoRxsaWJ2bWFjb3JlLnNvAAC00zAAJUw2AOZMNgDVUzYAzFk2AE0BMQCtGDEADiVBAYd/AGxpYnB0aHJlYWQuc28uMAACLzYPbGliYy5zby42AA==[/context]
2024-10-30T17:27:55.832Z warning vmware-dr[29772] [SRM@6876 sub=HttpConnectionPool-000000 opID=e92babb6-89fe-4235-bcb9-63cf21641aa5-test:fb7b:4589:c812:8a1e:b99d] Failed to get pooled connection; <cs p:00007fb6208e5830, TCP:VMware.support.loca:443>, (null), duration: 130389msec, N7Vmacore15SystemExceptionE(Connection timed out)
--> [context]zKq7AVECAAQAAAFuSQELdm13YXJlLWRyAADtoRxsaWJ2bWFjb3JlLnNvAAC00zAAJUw2AOZMNgDVUzYAzFk2AE0BMQCtGDEADiVBAYd/AGxpYnB0aHJlYWQuc28uMAACLzYPbGliYy5zby42AA==[/context]
2024-10-30T17:27:55.834Z error vmware-dr[29744] [SRM@6876 sub=IO.Http opID=e92babb6-89fe-4235-bcb9-63cf21641aa5-test:fb7b:4589:c812:2f0d:650d] User agent failed to send request; (null), N7Vmacore15SystemExceptionE(Connection timed out)
 

2024-10-30T17:27:55.842Z error vmware-dr[29739] [SRM@6876 sub=GuestOpsDomain.FileManager opID=e92babb6-89fe-4235-bcb9-63cf21641aa5-test:fb7b:4589:c812:8a1e:b99d] Failed to upload file to VM:'vm-11' by using uploadUrl:'https://VMware.support.local:443/guestFile?id=188&token=52e91855-09b0-14f1-a5fc-52606e54d4f1188'
--> N7Vmacore15SystemExceptionE Connection timed out (110)
--> [context]zKq7AVECAAQAAAFuSQELdm13YXJlLWRyAADtoRxsaWJ2bWFjb3JlLnNvAAC00zAAJUw2AOZMNgDVUzYAzFk2AE0BMQCtGDEADiVBAYd/AGxpYnB0aHJlYWQuc28uMAACLzYPbGliYy5zby42AA==[/context]
--> [backtrace begin] product: VMware vCenter Site Recovery Manager, version: 8.7.0, build: build-21589505, tag: vmware-dr, cpu: x86_64, os: linux, buildType: release
--> backtrace[03] libvmacore.so[0x001CA1ED]
--> backtrace[04] libvmacore.so[0x0030D3B4]
--> backtrace[05] libvmacore.so[0x00364C25]
--> backtrace[06] libvmacore.so[0x00364CE6]
--> backtrace[07] libvmacore.so[0x003653D5]
--> backtrace[08] libvmacore.so[0x003659CC]
--> backtrace[09] libvmacore.so[0x0031014D]
--> backtrace[10] libvmacore.so[0x003118AD]
--> backtrace[11] libvmacore.so[0x0041250E]
--> backtrace[12] libpthread.so.0[0x00007F87]
--> backtrace[13] libc.so.6[0x000F362F]
--> [backtrace end]


2024-10-30T17:27:55.845Z warning vmware-dr[29739] [SRM@6876 sub=Recovery opID=e92babb6-89fe-4235-bcb9-63cf21641aa5-test:fb7b:4589:c812:8a1e:b99d] [fb818380-dcf5-46cd-90c7-e17084706082.ipCust-protected-vm-220068451-IPCustomization]  UPLOAD_USER_FILE_0 Retrying (13 of 22) the guest operation after 60 seconds due to: N7Vmacore15SystemExceptionE Connection timed out (110)
--> [context]zKq7AVECAAQAAAFuSQELdm13YXJlLWRyAADtoRxsaWJ2bWFjb3JlLnNvAAC00zAAJUw2AOZMNgDVUzYAzFk2AE0BMQCtGDEADiVBAYd/AGxpYnB0aHJlYWQuc28uMAACLzYPbGliYy5zby42AA==[/context]
--> [backtrace begin] product: VMware vCenter Site Recovery Manager, version: 8.7.0, build: build-21589505, tag: vmware-dr, cpu: x86_64, os: linux, buildType: release
--> backtrace[03] libvmacore.so[0x001CA1ED]
--> backtrace[04] libvmacore.so[0x0030D3B4]
--> backtrace[05] libvmacore.so[0x00364C25]
--> backtrace[06] libvmacore.so[0x00364CE6]
--> backtrace[07] libvmacore.so[0x003653D5]
--> backtrace[08] libvmacore.so[0x003659CC]
--> backtrace[09] libvmacore.so[0x0031014D]
--> backtrace[10] libvmacore.so[0x003118AD]
--> backtrace[11] libvmacore.so[0x0041250E]
--> backtrace[12] libpthread.so.0[0x00007F87]
--> backtrace[13] libc.so.6[0x000F362F]
--> [backtrace end]
 

2024-10-30T15:34:12.510Z verbose vmware-dr[29780] [SRM@6876 sub=Recovery ctxID=39a014ae opID=752010ac-8c81-4b6f-b38e-f49838d0448b-test:c381:ce72:fd24:d1b4] [fb818380-dcf5-46cd-90c7-e17084706082.ipCust-protected-vm-219224654] Initializing IP customization for VM [vim.VirtualMachine:3080a9cc-cbd1-4e3f-b509-3e957f3671e8:vm-105696] with name Broadcom-1 [vm-11]

2024-10-30T16:14:50.999Z error vmware-dr[29738] [SRM@6876 sub=GuestOpsDomain.HttpClient-host-1485 opID=752010ac-8c81-4b6f-b38e-f49838d0448b-test:c381:ce72:fd24:d9fd:8896] Unable to upload file content to VM by using CS:<cs p:00007fb675df5ed0, TCP:VMware.support.local:443>, RequestPath: '/guestFile?id=193&token=52d3e8c3-ccf0-75b0-2c56-8717d4ebb618193'
2024-10-30T16:14:51.000Z error vmware-dr[29738] [SRM@6876 sub=GuestOpsDomain.FileManager opID=752010ac-8c81-4b6f-b38e-f49838d0448b-test:c381:ce72:fd24:d9fd:8896] Failed to upload file to VM:'vm-11' by using uploadUrl:'https://VMware.support.local:443/guestFile?id=193&token=52d3e8c3-ccf0-75b0-2c56-8717d4ebb618193'
2024-10-30T16:14:50.999Z info vmware-dr[29769] [SRM@6876 sub=IO.Http opID=752010ac-8c81-4b6f-b38e-f49838d0448b-test:c381:ce72:fd24:5c7d:8025] [527b9] Set user agent error; state: 1, (null), N7Vmacore17CanceledExceptionE(Operation was canceled)
2024-10-30T16:14:51.002Z warning vmware-dr[29749] [SRM@6876 sub=IO.Connection opID=752010ac-8c81-4b6f-b38e-f49838d0448b-test:c381:ce72:fd24:5c7d:8025] Failed to connect; <io_obj p:0x00007fb641b17f78, h:-1, <TCP '0.0.0.0 : 44528'>, <TCP '10.#.#.# : 443'>>, e: 104(Connection reset by peer), duration: 128298msec
2024-10-30T16:14:51.002Z info vmware-dr[29749] [SRM@6876 sub=IO.Connection opID=752010ac-8c81-4b6f-b38e-f49838d0448b-test:c381:ce72:fd24:5c7d:8025] Failed to shutdown socket; <io_obj p:0x00007fb641b17f78, h:-1, <TCP '0.0.0.0 : 44528'>, <TCP '10.#.#.# : 443'>>, e: 9(shutdown: Bad file descriptor)
2024-10-30T16:14:51.002Z warning vmware-dr[29749] [SRM@6876 sub=IO.Connection opID=752010ac-8c81-4b6f-b38e-f49838d0448b-test:c381:ce72:fd24:5c7d:8025] Aborting connection attempt: unexpected state; <resolver p:0x00007fb67d797af0, 'VMware.support.local:443', next:(null)>, (null)(Socked was closed manually), last e: 104(Connection reset by peer)
2024-10-30T16:14:51.002Z warning vmware-dr[29749] [SRM@6876 sub=HttpConnectionPool-000000 opID=752010ac-8c81-4b6f-b38e-f49838d0448b-test:c381:ce72:fd24:5c7d:8025] Failed to get pooled connection; <cs p:00007fb612a53df0, TCP:VMware.support.local:443>, (null), duration: 128300msec, N7Vmacore17CanceledExceptionE(Operation was canceled)

2024-10-30T20:32:38.503Z warning vmware-dr[01468] [SRM@6876 sub=IO.Connection] Failed to connect; <io_obj p:0x00007fb659dc2158, h:25, <TCP '10.4.201.117 : 43218'>, <TCP '10.7.30.54 : 443'>>, e: 111(Connection refused), duration: 39msec
2024-10-30T20:32:38.503Z verbose vmware-dr[01468] [SRM@6876 sub=IO.Connection] Attempting connection; <resolver p:0x00007fb60ddee180, 'srm09-024-dfw3.rvi.rax.io:443', next:(null)>, last e: 111(Connection refused)
2024-10-30T20:32:38.503Z warning vmware-dr[01468] [SRM@6876 sub=HttpConnectionPool-000001] Failed to get pooled connection; <cs p:00007fb64802a7e0, TCP:srm09-024-dfw3.rvi.rax.io:443>, (null), duration: 42msec, N7Vmacore15Sys
temExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)
--> [context]zKq7AVECAAQAAAFuSQELdm13YXJlLWRyAADtoRxsaWJ2bWFjb3JlLnNvAAC00zAAJUw2AOZMNgDVUzYAzFk2AE0BMQCtGDEADiVBAYd/AGxpYnB0aHJlYWQuc28uMAACLzYPbGliYy5zby42AA==[/context]
2024-10-30T20:32:38.504Z info vmware-dr[01468] [SRM@6876 sub=IO.Http] Set user agent error; state: 1, (null), N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall
 is rejecting connections.)
--> [context]zKq7AVECAAQAAAFuSQELdm13YXJlLWRyAADtoRxsaWJ2bWFjb3JlLnNvAAC00zAAJUw2AOZMNgDVUzYAzFk2AE0BMQCtGDEADiVBAYd/AGxpYnB0aHJlYWQuc28uMAACLzYPbGliYy5zby42AA==[/context]
2024-10-30T20:32:38.504Z error vmware-dr[01468] [SRM@6876 sub=IO.Http] User agent failed to send request; (null), N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a fire
wall is rejecting connections.)
--> [context]zKq7AVECAAQAAAFuSQELdm13YXJlLWRyAADtoRxsaWJ2bWFjb3JlLnNvAAC00zAAJUw2AOZMNgDVUzYAzFk2AE0BMQCtGDEADiVBAYd/AGxpYnB0aHJlYWQuc28uMAACLzYPbGliYy5zby42AA==[/context]
2024-10-30T20:32:38.505Z warning vmware-dr[27046] [SRM@6876 sub=RemoteDrServer connID=dr-admin-bab8] Failed to connect:
--> N7Vmacore15SystemExceptionE Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections. (111)
--> [context]zKq7AVECAAQAAAFuSQELdm13YXJlLWRyAADtoRxsaWJ2bWFjb3JlLnNvAAC00zAAJUw2AOZMNgDVUzYAzFk2AE0BMQCtGDEADiVBAYd/AGxpYnB0aHJlYWQuc28uMAACLzYPbGliYy5zby42AA==[/context]
--> [backtrace begin] product: VMware vCenter Site Recovery Manager, version: 8.7.0, build: build-21589505, tag: vmware-dr, cpu: x86_64, os: linux, buildType: release
--> backtrace[03] libvmacore.so[0x001CA1ED]
--> backtrace[04] libvmacore.so[0x0030D3B4]
--> backtrace[05] libvmacore.so[0x00364C25]
--> backtrace[06] libvmacore.so[0x00364CE6]
--> backtrace[07] libvmacore.so[0x003653D5]
--> backtrace[08] libvmacore.so[0x003659CC]
--> backtrace[09] libvmacore.so[0x0031014D]
--> backtrace[10] libvmacore.so[0x003118AD]
--> backtrace[11] libvmacore.so[0x0041250E]
--> backtrace[12] libpthread.so.0[0x00007F87]
--> backtrace[13] libc.so.6[0x000F362F]
--> [backtrace end]

Environment

VMware Live Site Recovery
Site Recovery Manager 

Cause

The required ports for IP Customization are not open. 

Resolution

Run a curl test from the SRM Appliance to the host to check if the ports are open. 

root@SRM [ ]# curl -v telnet://ESXi.support.local:443
*   Trying 10.#.#.#:443...

If the test does not complete, press Ctrl+C to quit. 

Network Ports for Site Recovery Manager

Open the necessary ports on the Firewall or add rules to NSX to allow traffic between the SRM Appliance and the hosts. 

443	HTTPS	Site Recovery Manager on the recovery site	Recovery site ESXi host.	Traffic from the Site Recovery Manager Server on the recovery site to ESXi hosts when recovering or testing virtual machines with configured IP customization, or callout commands on recovered virtual machines.
902	TCP and UDP	Site Recovery Manager Server on the recovery site.	Recovery site ESXi host.	Traffic from the Site Recovery Manager Server on the recovery site to ESXi hosts when recovering or testing virtual machines with IP customization, with configured callout commands on recovered virtual machines, or that use raw disk mapping (RDM). All NFC traffic for updating or patching the VMX files of virtual machines that are replicated using vSphere Replication use this port