vSphere Replication HBR service takes a long time to start

Products

• VMware Live Recovery
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• vSphere Replication HBR service takes a long time to start.
• ​In /var/log/vmware/hbrsrv.log you see entries similar to:

Heartbeat handler detected dead connection for host: host-6627
HbrError stack:
[0] Exception Vmacore::InvalidStateException: No connection (host=host-6627)2017-04-27T20:57:45.928Z [7FDAC6F3C700 warning 'Default'] Failed to connect socket; <io_obj p:0x0000000005403810, h:573, <TCP '0.0.0.0:0'>, <TCP 'X.X.X.X:80'>>, e: system:111(Connection refused)

Environment

VMware vSphere Replication 6.0.x
VMware vSphere Replication 5.5.x
VMware vSphere Replication 6.5.x
VMware vSphere Replication 5.1.x
VMware vSphere Replication 5.8.x

Cause

During startup, vSphere Replication hbrsrv needs to try to connect to all hosts in the vCenter inventory. If customer has a large environment and a single hosts cannot connect to hbrsrv, it will take long time to loop through them. Network firewall may add unnecessary time to HBR service startup.
 
If an ESXi host is removed from the VC inventory while the vSphere Replication hbrsrv is offline that will also interfere with the startup of the service as vSphere Replication hbrsrv still has an ESXi entry for that host in its database.

Resolution

To resolve this issue,

1. Investigate network firewall as port 80 is required to be open between the vSphere Replication server and ESXi host (intra-site). For more information see  Port numbers that must be open for vSphere Replication 5.8.x and 6.x.
2. If vSphere Replication server is attempting to register and connect to a host which no longer exists in the vCenter inventory, an edit of the hbrsrv.db may be required. Contact VMware support if this is encountered.