http://127.0.0.1:9286/sdk invocation failed with java.net.SocketTimeoutException
search cancel

http://127.0.0.1:9286/sdk invocation failed with java.net.SocketTimeoutException

book

Article ID: 312692

calendar_today

Updated On:

Products

VMware Live Recovery

Issue/Introduction

Symptoms:


  1. Host & vCenter/VRA appliances are syncing with the same NTP but you continuously hit the same error
  2. Updating DNS from VAMI will not work because hbrsvc & hms service must be in running state for it to take effect
  3. You cannot update DNS from CLI thru vami_config_net
  4. Switching from NTP to HOST within the time synchronization settings of vCenter & VR will not work (Even if the hosts are syncing with the same NTP)
  5. Rebooting vCenter & VRA does not help
  6. vCenter Server with an external Platform Services Controller to a vCenter Server with an embedded Platform Services Controller convergence causes this error. 
  7. VR VAMI displays this message when you click on summary page - http://127.0.0.1:9286/sdk invocation failed
  8. This error occurs when the ovfEnv.xml, HMS configuration file & VRMS DB does not get updated with the correct vCenter URL & thumbprints. 
You are bound to hit this error when you RECONFIGURE VR after performing an upgrade from 8.x to 8.4 etc.

ERROR
http://127.0.0.1:9286/sdk invocation failed with "java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-165 [ACTIVE]"
30,000 milliseconds timeout on connection http-outgoing-165 [ACTIVE]
Operation ID: 7fde2086-4972-49ca-903e-9a403ac75b7d
4/15/21, 8:49:08 AM -0400

Also look for these messages below, if its not DNS causing the problem.

/var/log/vmware/drconfigui/dr-config.log :

2023-01-27 20:03:22,210 [srm-reactive-thread-5] WARN  com.vmware.dr.configservice.summary.IsReconfigureRequiredHandler bd06a10d-bf5c-4193-81ab-8da29cd4509d getReconfigureRequired - DrRequestHandlerError:

com.vmware.vim.vmomi.client.exception.ConnectionException: http://127.0.0.1:9286/sdk invocation failed with "java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-14 [ACTIVE]"
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:261)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.setResponseError(HttpExchangeBase.java:321)
        at com.vmware.vim.vmomi.client.http.impl.HttpAsyncExchange$1$1.invokeWithinScope(HttpAsyncExchange.java:131)
        at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:24)
        at com.vmware.dr.ui.tools.utilities.ThreadContext.lambda$wrap$10(ThreadContext.java:55)
        at com.vmware.dr.ui.tools.utilities.ThreadContext.execute(ThreadContext.java:209)
        at com.vmware.dr.ui.tools.utilities.ThreadContext.execute(ThreadContext.java:185)
        at com.vmware.dr.ui.tools.utilities.ThreadContext.setupContext(ThreadContext.java:76)
        at com.vmware.dr.ui.tools.utilities.ThreadContext.setupContext(ThreadContext.java:105)
        at com.vmware.dr.ui.tools.utilities.ExecutorUtils.lambda$wrap$6(ExecutorUtils.java:36)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-14 [ACTIVE]
        at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:381)
        at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
        at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
        at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:263)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:492)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:213)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
        ... 1 more


/var/log/vmware/dr/drconfig.log :  

2023-01-27T20:02:52.179Z info drconfig[00774] [SRM@6876 sub=DrConfigConfigurationManager opID=bd06a10d-bf5c-4193-81ab-8da29cd4509d-isReconfigureRequired] ReadVrmsCurrentConfig: Reading VRMS Configuration.
2023-01-27T20:02:52.179Z verbose drconfig[00774] [SRM@6876 sub=Default opID=bd06a10d-bf5c-4193-81ab-8da29cd4509d-isReconfigureRequired] Loading configuration /opt/vmware/hms/conf/hms-configuration.xml
2023-01-27T20:02:52.180Z info drconfig[00774] [SRM@6876 sub=Default.SiteConnections opID=bd06a10d-bf5c-4193-81ab-8da29cd4509d-isReconfigureRequired] Compatible VMODL version found: 'vim.version.version11/internalvim25/6.5'
2023-01-27T20:02:52.180Z verbose drconfig[00774] [SRM@6876 sub=Default.SiteConnections opID=bd06a10d-bf5c-4193-81ab-8da29cd4509d-isReconfigureRequired] Event broadcasted
2023-01-27T20:02:52.180Z info drconfig[00774] [SRM@6876 sub=Default.SiteConnections opID=bd06a10d-bf5c-4193-81ab-8da29cd4509d-isReconfigureRequired] Compatible VMODL version found: 'dr.version.version14/internaldr0/8.2'
2023-01-27T20:02:52.180Z verbose drconfig[00774] [SRM@6876 sub=Default.SiteConnections opID=bd06a10d-bf5c-4193-81ab-8da29cd4509d-isReconfigureRequired] Event broadcasted
2023-01-27T20:02:52.180Z info drconfig[00774] [SRM@6876 sub=Default.SiteConnections opID=bd06a10d-bf5c-4193-81ab-8da29cd4509d-isReconfigureRequired] Compatible VMODL version found: 'hms.version.version12/internalhmssrv/8.0'
2023-01-27T20:02:52.180Z verbose drconfig[00774] [SRM@6876 sub=Default.SiteConnections opID=bd06a10d-bf5c-4193-81ab-8da29cd4509d-isReconfigureRequired] Event broadcasted

2023-01-27T20:02:52.180Z verbose drconfig[00774] [SRM@6876 sub=Default.SiteConnections opID=bd06a10d-bf5c-4193-81ab-8da29cd4509d-isReconfigureRequired] SiteConnections created for 'https://1
72.x.x.22:443/lookupservice/sdk' with thumbprint '1C:D4:B0:86:DD:90:C4:C4:7D:FD:04:D0:xx:xx:xx:xx:xx:xx:xx:72:94:B9:1D:C5:19:54:78:55:96:FF:A1:FB'
2023-01-27T20:02:52.184Z verbose drconfig[00774] [SRM@6876 sub=vmomi.soapStub[12] opID=bd06a10d-bf5c-4193-81ab-8da29cd4509d-isReconfigureRequired] Connected to host <cs p:00007f964800f390, TCP:172.x.x.22:443> using internallookup/2.0 (lookup.version.version2)
2023-01-27T20:03:13.780Z warning drconfig[01363] [SRM@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f9600004098, h:10, <TCP '172.x.x.7 : 50936'>, <TCP '172.x.x.22 : 443'>>, e: 110(Connection timed out)
2023-01-27T20:03:13.780Z verbose drconfig[01362] [SRM@6876 sub=vmomi.soapStub[11] opID=8f3c8409-f5b3-4096-9a87-2f6eab6999d2-isReconfigureRequired] Resetting stub adapter for server <cs p:00007f961c019400, TCP:172.x.x.22:443> : Closed
2023-01-27T20:03:13.780Z error drconfig[01362] [SRM@6876 sub=DrConfigConfigurationManager opID=8f3c8409-f5b3-4096-9a87-2f6eab6999d2-isReconfigureRequired] Failed to obtain VC instance id by URL.
2023-01-27T20:03:13.781Z error drconfig[01362] [SRM@6876 sub=Default opID=8f3c8409-f5b3-4096-9a87-2f6eab6999d2-isReconfigureRequired] Failed to send response to the client
--> N7Vmacore11IOExceptionE System exception while trasmitting HTTP Response:
--> error id = 32
-->
--> [context]zKq7AVECAAQAAEnqDQEMZHJjb25maWcAAL3EG2xpYnZtYWNvcmUuc28AAEMOFwC7WyMAlVwjAY6OGmxpYnZtb21pLnNvAAKYCgZsaWJkci12bW9taS5zbwAC7xQGAO3/LgAyHC8AfjJAA4d/AGxpYnB0aHJlYWQuc28uMAAEvzUPbGliYy5zby42AA==[/context]
--> [backtrace begin] product: VMware vCenter Site Recovery Manager, version: 8.4.0, build: build-17689161, tag: drconfig, cpu: x86_64, os: linux, buildType: release
--> backtrace[03] libvmacore.so[0x001BC4BD]
--> backtrace[04] libvmacore.so[0x00170E43]
--> backtrace[05] libvmacore.so[0x00235BBB]
--> backtrace[06] libvmacore.so[0x00235C95]
--> backtrace[07] libvmomi.so[0x001A8E8E]
--> backtrace[08] libdr-vmomi.so[0x00060A98]
--> backtrace[09] libdr-vmomi.so[0x000614EF]
--> backtrace[10] libvmacore.so[0x002EFFED]
--> backtrace[11] libvmacore.so[0x002F1C32]
--> backtrace[12] libvmacore.so[0x0040327E]
--> backtrace[13] libpthread.so.0[0x00007F87]
--> backtrace[14] libc.so.6[0x000F35BF]
--> [backtrace end]
2023-01-27T20:03:23.770Z warning drconfig[01362] [SRM@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f9604002d28, h:12, <TCP '172.x.x.7 : 50950'>, <TCP '172.x.x.22 : 443'>>, e: 110(Connection timed out)
2023-01-27T20:03:23.770Z verbose drconfig[00774] [SRM@6876 sub=vmomi.soapStub[12] opID=bd06a10d-bf5c-4193-81ab-8da29cd4509d-isReconfigureRequired] Resetting stub adapter for server <cs p:00007f964800f390, TCP:172.x.x.22:443> : Closed
2023-01-27T20:03:23.770Z error drconfig[00774] [SRM@6876 sub=DrConfigConfigurationManager opID=bd06a10d-bf5c-4193-81ab-8da29cd4509d-isReconfigureRequired] Failed to obtain VC instance id by URL.

Environment

VMware vSphere Replication 8.x

Cause


The connection to vCenter is taking a long time and the socket is timing out before a connection to vCenter is established because the first 2 DNS servers are decommissioned. Henceforth, when the appliance tries to communicate it times out before it can hit the 3rd DNS within 30 seconds.



From the screenshot, 192.x.x.12 and 192.x.x.13 DNS servers have been decommissioned and 192.x.x.2 is the one that’s active & this is affecting the RECONFIGURE process because the appliance always hits the first DNS server found before moving on to others.

Resolution

  1. Enable SSH from the replication appliance VAMI
  2. Execute the following command to update the proper DNS
       /opt/vmware/share/vami/vami_set_dns  -d domain  -s searchpath DNS_Server_1 DNS_Server_2
  1. Replace domain, searchpath, DNS_Server_1, DNS_Server_2 with the correct values. The domain, searchpath, and second DNS Server are optional and can be skipped.
Example:

The following command will only set a domain and one DNS server: /opt/vmware/share/vami/vami_set_dns  -d vr.local 192.x.x.2

The following command will set all fields: /opt/vmware/share/vami/vami_set_dns  -d vr.local  -s vr.local 192.x.x.2 192.x.x.3
  1. Verify the DNS changes by running the following command: resolvectl status
  2. RECONFIGURE should work now

Continue with the CHECKS below, if fixing DNS does not resolve the problem. 
Before performing the checks below, its mandatory to ensure vCenter is working in a healthy condition. 

NOTE: Please take snapshots on vCenter & VR 

1. External PSC, if any must be properly decommissioned. (This has to be validated when a convergence of vCenter Server with an external Platform Services Controller to a vCenter Server with an embedded Platform Services Controller occurs)
2. SSL trust mismatch issues in the lookup service must be fixed by running the LSDoctor tool (Use the switches -l, -s & -t)
3. All vCenter services must be restarted in the SSO site or ELM (Enhanced Linked Mode)

Using the 'lsdoctor' Tool (80469) 

Refer to this KB to get thumbprints - Finding SHA-1 and SHA-256 thumbprints for ESXi, vCenter, vSphere Replication and Site Recovery Manager appliances (82548)

CHECK # 1 : Check and update the ovfEnv.xml & hms-configuration.xml files. 

The following commands must be run in 'root' mode after logging in to VR Appliance. 

1. Stop HMS & HBRSRV services 

root@VRMSPR [ ~ ]# systemctl stop hms
root@VRMSPR [ ~ ]# systemctl stop hbrsrv

2. cd /opt/vmware/etc/vami/

root@VRMSPR [ /opt/vmware/etc/vami]# ls -lah
total 28K
drwxr-xr-x  3 root root 4.0K Jan 28 21:16 .
drwxr-xr-x 15 root root 4.0K Apr 28  2022 ..
drwxr-xr-x  2 root root 4.0K Apr 28  2022 flags
-rw-r--r--  1 root root 2.1K Jan 28 21:16 ovfEnv.xml
-rw-r--r--  1 root root 2.2K Jul  6  2022 ovfEnv.xml.bkp
-rw-r--r--  1 root root  166 Feb 26  2021 vami.xml
-rw-r--r--  1 root root  129 Apr 28  2022 vami_ovf_info.xml

3. Edit the ovfEnv.xml file & modify the highlighted lines. I recommend to power OFF & power ON the VR appliance once or twice to see whether VR automatically fetches this information from vCenter (most times, it will). If not, we will need to update this file manually. 

evs:URL > vCenter URL 
evs:IP  > vCenter IP 
evs:Address > vCenter URL 

   </PropertySection>
   <ve:EthernetAdapterSection>
      <ve:Adapter ve:mac="00:50:##:##:##:59" ve:network="REPLICACION" ve:unitNumber="7"/>
   </ve:EthernetAdapterSection>
<ve:vServiceEnvironmentSection xmlns:ve="http://www.vmware.com/schema/ovfenv" xmlns:evs="http://www.vmware.com/schema/vservice/ExtensionVService" ve:bound="true" ve:id="installation" ve:type="com.vmware.vservice.extension">
    <evs:GuestApi>
        <evs:URL>https://vcenter-fqdn/vsm/extensionService</evs:URL>
        <evs:Token>93bdb6a3505b9aea6ac019b75b0b5859f5f4c227</evs:Token>
        <evs:X509Thumbprint>4b:c9:0b:75:20:23:d6:xx:xx:xx:xx:xx:xx:xx:xx:dc:eb:9a:f5:66</evs:X509Thumbprint>
    </evs:GuestApi>
    <evs:VCenterApi>
        <evs:IP>172.##.###.20</evs:IP>
        <evs:X509Thumbprint>4b:c9:0b:75:20:23:d6:xx:xx:xx:xx:xx:xx:xx:xx:dc:eb:9a:f5:66</evs:X509Thumbprint>
        <evs:HttpPort>80</evs:HttpPort>
        <evs:HttpsPort>443</evs:HttpsPort>
        <evs:SelfMoRef>VirtualMachine:vm-106</evs:SelfMoRef>
        <evs:Address>vcenter-fqdn</evs:Address>
    </evs:VCenterApi>
</ve:vServiceEnvironmentSection>
</Environment>

If you are making changes to ovfEnv.xml file, run the following command for it to take effect. This command also helps in setting the evs:Token if its changed.

vmware-rpctool "info-set guestinfo.ovfEnv `cat /opt/vmware/etc/vami/ovfEnv.xml`"

This command can also be used to check the ovfEnv.xml file : vmware-rpctool "info-get guestinfo.ovfEnv"

4. Edit the hms-configuration.xml & update the items highlighted below. 

Path: /opt/vmware/hms/conf/hms-configuration.xml

   <!-- Local vCenter Server settings -->
   <hms-localvc-address>VCSA67PR.vr.local</hms-localvc-address>
   <hms-localvc-port>80</hms-localvc-port>
   <hms-localvc-thumbprint>12:6A:F7:33:15:AA:6B:24:4D:34:B3:3D:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:2A:D2:EA:FE:A0:56:41</hms-localvc-thumbprint>
   <hms-localvc-adminmail>[email protected]</hms-localvc-adminmail>

   <!-- LookupService connection settings -->
   <hms-ls-url>https://VCSA67PR.vr.local:443/lookupservice/sdk</hms-ls-url>
   <hms-ls-thumbprint>12:6A:F7:33:15:AA:6B:24:4D:34:B3:3D:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:2A:D2:EA:FE:A0:56:41</hms-ls-thumbprint>

5. Start HMS & HBRSRV services 

root@VRMSPR [ ~ ]# systemctl start hms
root@VRMSPR [ ~ ]# systemctl start hbrsrv


If the above steps do not resolve the issue, please contact Broadcom Support for further investigation.

Additional Information

Powered by Wolken Software