vSphere replication addon disconnected in SRM UI

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

1. VR add-on server shows disconnected
2. Got error if reconnect or resync replicaton on it
   Cannot establish a TCP connection to server at 'X.X.X.X:8123'. Details: 'https://X.X.X.X:8123/ invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to X.X.X.X:8123 [/X.X.X.X] failed: Connection refused (Connection refused)"'.
   
3. 'sytemctl status hbrsrv' shows hbrsrv service restarts repeatedly
4. hbrsrv crashes with trace
   Panic: Resource temporarily unavailable

Environment

VMware vSphere Replication 9.0.2

Cause

There are high number replication in not active status to recover, which leads hbrsrv out of resource.

Resolution

1. Take a snapshot of the VM
2. SSH to the appliance
3. Edit /usr/lib/systemd/system/hbrsrv.service and add "TasksMax=infinity" line as shown below.
[Service]
Type=forking
EnvironmentFile=/etc/hbrsrv-init.config
User=hbrsrv
Group=vmware
PermissionsStartOnly=true
TasksMax=infinity
4. Run the commands below :
systemctl daemon-reload
systemctl restart hbrsrv

If the above action plan can not resolve the issue, please collect dump with command toggle-coredumps.sh
Once a new dump under /var/core is created, please run toggle-coredumps.sh again to stop creating dump.

Additional Information

Refer Troubleshooting a disconnected vSphere Replication Appliance OR vSphere replication server randomly connects and disconnects in SRM UI