vSphere Replication server is disconnected - Cannot establish a TCP connection to server
Article ID: 312710
Updated On:
Products
Issue/Introduction
Symptoms:
VR server is disconnected in SRM UI
Reconnecting it throws the error -
ERROR
Operation Failed
Cannot establish a TCP connection to server at '10.X.X.X:8123'. Details: 'https://10.X.X.X:8123 invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to 10.X.X.X:8123 [/10.X.X.X] failed: Connection refused (Connection refused)"'.
/opt/vmware/hms/logs/hms.log :
2024-01-26 15:09:09.042 ERROR hms.net.hbr.ping.svr.52d8c50d-9d4c-2c51-2d0f-5c488ace3100 [hms-ping-scheduled-thread-8] (..net.impl.VmomiPingConnectionHandler) [operationID=d82ac874-74b9-43bc-9b68-32bcee38c3dd-HMS-PING---Ping Thread for session key: N/A and vmomi session: null and server: 10.X.X.X:8123] | Ping for server 10.X.X.X:8123 for session: N/A failed: com.vmware.vim.vmomi.client.exception.ConnectionException: https://10.X.X.X:8123/ invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to 10.X.X.X:8123 [/10.X.X.X] failed: Connection refused (Connection refused)" : https://10.X.X.X:8123/ invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to 10.X.X.X:8123 [/10.X.X.X] failed: Connection refused (Connection refused)"
2024-01-26 15:10:49.048 ERROR hms.net.hbr.ping.svr.52d8c50d-9d4c-2c51-2d0f-5c488ace3100 [hms-ping-scheduled-thread-7] (..net.impl.VmomiPingConnectionHandler) [operationID=d82ac874-74b9-43bc-9b68-32bcee38c3dd-HMS-PING---Ping Thread for session key: N/A and vmomi session: null and server: 10.X.X.X:8123] | Ping for server 10.167.52.162:8123 for session: N/A failed: com.vmware.vim.vmomi.client.exception.ConnectionException: https://10.X.X.X:8123/ invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to 10.X.X.X:8123 [/10.X.X.X] failed: Connection refused (Connection refused)" : https://10.X.X.X:8123/ invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to 10.X.X.X:8123 [/10.X.X.X] failed: Connection refused (Connection refused)"
2024-01-26 15:11:03.784 TRACE hms.net.hbr.ping.svr.5206d433-d89c-31fb-f81e-039bc557722a [hms-ping-scheduled-thread-3] (..net.impl.VmomiPingConnectionHandler) [operationID=6d84cf34-0618-4294-a9ca-d54ed10b3ece-HMS-PING---Ping Thread for session key: N/A and vmomi session: A7F0C05C0137C07A5C606DB9622279528A7E980CFCAB0C98B6BC6187AF6175DF9C675BD74CC24B98F1B6B935F1FFFEC860A1B8DEE39BBB4E54E8649C06960B04 and server: 127.0.0.1:8123] | Session: N/A on server '127.0.0.1:8123' pinged successfully
2024-01-26 15:11:09.049 ERROR hms.net.hbr.ping.svr.52d8c50d-9d4c-2c51-2d0f-5c488ace3100 [hms-ping-scheduled-thread-4] (..net.impl.VmomiPingConnectionHandler) [operationID=d82ac874-74b9-43bc-9b68-32bcee38c3dd-HMS-PING---Ping Thread for session key: N/A and vmomi session: null and server: 10.X.X.X:8123] | Ping for server 10.X.X.X:8123 for session: N/A failed: com.vmware.vim.vmomi.client.exception.ConnectionException: https://10.X.X.X:8123/ invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to 10.X.X.X:8123 [/10.X.X.X] failed: Connection refused (Connection refused)" : https://10.X.X.X:8123/ invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to 10.X.X.X:8123 [/10.X.X.X] failed: Connection refused (Connection refused)"
/var/log/vmware/hbrsrv.log :
2024-01-26T15:39:53.439Z panic hbrsrv[04338] [Originator@6876 sub=Main] HbrError stack:
2024-01-26T15:39:53.439Z panic hbrsrv[04338] [Originator@6876 sub=Main] [0] Could not compute a reasonable expiration time for private key '/etc/vmware/ssl/hbrsrv_broker_private.pem'
2024-01-26T15:39:53.439Z panic hbrsrv[04338] [Originator@6876 sub=Main] [1] Now: 2024-01-26T15:39:53.439392Z Creation time: 2023-05-04T19:02:01Z
2024-01-26T15:39:53.439Z panic hbrsrv[04338] [Originator@6876 sub=Default]
-->
--> Panic: Couldn't initialize broker token manager crypto!
--> Backtrace:
--> [backtrace begin] product: VMware vSphere Replication Server, version: 8.0.0, build: build-21111478, tag: hbrsrv, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] hbrsrv-bin[0x00B9D53F]
--> backtrace[01] hbrsrv-bin[0x00B938C2]
--> backtrace[02] hbrsrv-bin[0x00CD4677]
--> backtrace[03] hbrsrv-bin[0x00CD4791]
--> backtrace[04] hbrsrv-bin[0x009816C5]
--> backtrace[05] hbrsrv-bin[0x008EB963]
--> backtrace[06] hbrsrv-bin[0x007C45B2]
--> backtrace[07] libc.so.6[0x00022B87]
--> backtrace[08] hbrsrv-bin[0x008906E1]
--> [backtrace end]
Environment
Cause
hbrsrv can't regenerate private key due to which it is going into panic mode.
/var/log/vmware/hbrsrv-generate-keypair.log :
Generating private key /etc/vmware/ssl/hbrsrv_broker_private.pem_tmp with 2048 bits
Generating RSA private key, 2048 bit long modulus
..+++++.............................................................+++++
unable to write 'random state'
e is 65537 (0x10001)
Generating public key /etc/vmware/ssl/hbrsrv_broker_public.pem_tmp
writing RSA key
Done.
Resolution
1. Take a normal snapshot.
2. Check if you can generate keys manually by running this command -
hbrsrv-generate-keypair.sh /etc/vmware/ssl/hbrsrv_broker_private.pem /etc/vmware/ssl/hbrsrv_broker_public.pem 2048
3. If you can generate keys manually go to /etc/vmware/ssl
root@vrmspr [ /etc/vmware/ssl ]# ls -lah
total 24K
drwxrwxr-x 2 root vmware 4.0K Feb 1 13:59 .
drwxrwxr-x 3 root vmware 4.0K Jan 26 16:03 ..
-rw-r--r-- 1 hbrsrv vmware 1.4K May 4 2023 hbrsrv.crt
-rw-r--r-- 1 hbrsrv vmware 1.7K May 4 2023 hbrsrv.key
-r-------- 1 root root 1.7K Feb 1 13:59 hbrsrv_broker_private.pem
-rw-r--r-- 1 root root 451 Feb 1 13:59 hbrsrv_broker_public.pem
3. Create a backup of hbrsrv_broker_private.pem & hbrsrv_broker_public.pem and move it to a backup directory
Delete the files -
root@vrmspr [ /etc/vmware/ssl ]# rm -r hbrsrv_broker_private.pem
root@vrmspr [ /etc/vmware/ssl ]# rm -r hbrsrv_broker_public.pem
NOTE: hbrsrv will automatically regenerate them upon deletion.
4. systemctl restart hbrsrv
Now, the replication server should reconnect.
Additional Information
Feedback
