vSphere Replication sending large number of HTTPS requests to envoy proxy causing hostd to crash - remote https connections exceed max allowed

Products

VMware Live Recovery VMware vSphere ESXi

Issue/Introduction

In vCenter, the Host intermittently shows to be in an unresponsive state.
Host becomes disconnected in vCenter.
Host client cannot be accessed.
While reconnecting/adding a host to vCenter, it fails with the following error:
"Cannot contact the specified host (hostname.example.com). The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding."
Restarting all services on the host (services.sh restart) makes the host stable for some time before going into a not responding state again.
Powering OFF the vSphere Replication appliance stops the HTTPS requests made by VR, thereby bringing the host to a normal state.
vCenter tasks are filling out with the error: A generic error occurred in the vSphere Replication Management Server. Exception details: 'Unexpected status code: 503'.

/var/run/log/envoy-access.log:

YYYY-MM-DD In(166) envoy-access[2098958]: POST /hbr HTTP/1.1 200 via_upstream - 586 455 1 1 0 ##.#.#.#:50190 TLSv1.2 ##.#.#.#:443 - - /var/run/vmware/proxy-hbr "########-####-####-####-############-HMS-PING" "Fetch"

Log shows a large number of HMS-PING calls received by VRMS.

/opt/vmware/hms/logs/hms.log:

YYYY-MM-DD 20:#:#.# TRACE hms.net.hbr.ping.svr.########-####-####-####-############ [hms-ping-scheduled-thread-0] (..net.impl.VmomiPingConnectionHandler) [operationID=########-####-####-####-############-HMSINT-5641, operationID=########-####-####-####-############-HMS-PING] | Session: N/A on server '##.#.#.#:443/hbr' pinged successfully

/var/run/log/envoy.log:

YYYY-MM-DD In(166) envoy[2098941]: "YYYY-MM-DD warning envoy[2099266] [Originator@6876 sub=filter] [C18705] closing connection TCP<##.#.#.#:33556, ##.#.#.#:443>"
YYYY-MM-DD In(166) envoy[2098941]: "YYYY-MM-DD warning envoy[2099264] [Originator@6876 sub=filter] [C18706] remote https connections exceed max allowed: 128"

Environment

VMware vSphere Replication 8.x, 9.0
VMware vCenter 8.x

Cause

vSphere Replication causes the hosts to become unresponsive due to an authentication issue between hms and hbrsrvuw in ESXi. This can randomly affect any ESXi host in the vCenter inventory.

The root cause is that getServers().registerHbrServer(hbrServerData) starts a new ReconnectingPing, but it is not cleaned up when an exception occurs. Therefore, the leaked ReconnectingPing continuously increases every minute. The hbrsrvuw ReconnectingPing leaks when it is unable to connect to hbrsrvuw, causing the ESXi host to become unavailable.

Resolution

This issue has been fixed in vSphere Replication 9.0.1 | 25 JUN 2024 | Build 24037980 | Download

Please upgrade to this version at the earliest if you are running on an older version of the vSphere Replication appliance to avoid this issue. It has also been noticed that hosts still go into a 'Not Responding' state despite upgrading the appliance to this version. In such cases, please only disable scale-out-mode on the VRMS at both sites.

Workaround:

The purpose of the workaround is to REMOVE all host-based replication servers from the 'Replication Servers' tab in the SRM UI. This will stop HMS from pinging hbrsrvuw in the ESXi host.

🚨 Warning: This fix has to be applied on replication servers running version 8.8.X at both source and target sites. We are occasionally seeing 9.x releases also affected, so please apply the fix mentioned here.

SSH to the vSphere Replication appliance. Run the command: systemctl stop hms

Edit /opt/vmware/hms/conf/hms-configuration.xml and change scale-out-mode to false.

<scale-out-mode>false</scale-out-mode>
<!--
Timeout to wait before tagging hbrsrv as decommissioned due to maintenance mode.
At the moment set to 0 since replications are auto-released on hbrsrvuw when ESX host enters MM.
-->

NOTE: In vSphere replication 9.0 and higher, this option is not compatible with enhanced replication. If scale-out-mode is set to true (enhanced replication is enabled) and hms-embedded-hbr is set to false, an error with the phrase "Unable to connect to the HBR Management Server" will be observed.

Login to the VR Database: /opt/vmware/hms/bin/embedded_db_connect.sh
Run the SQL commands:

NOTE: Please run these commands in the order they are mentioned below.

Query the hbrsrvuw which does not have replications from hbrtagentity & hbrserverentity tables.

select hbrserver_movalue from hbrtagentity where hbrserver_movalue IN (select movalue from hbrserverentity where vsrv_port = 443 AND NOT EXISTS (select hbrserver_movalue from secondarygroupentity where hbrserverentity.movalue = secondarygroupentity.hbrserver_movalue));

select hbrservername,movalue from hbrserverentity where vsrv_port = 443 AND NOT EXISTS (select hbrserver_movalue from secondarygroupentity where hbrserverentity.movalue = secondarygroupentity.hbrserver_movalue);

Delete the hbrsrvuw which does not have replications from hbrtagentity & hbrserverentity tables.

delete from hbrtagentity where hbrserver_movalue IN (select movalue from hbrserverentity where vsrv_port = 443 AND NOT EXISTS (select hbrserver_movalue from secondarygroupentity where hbrserverentity.movalue = secondarygroupentity.hbrserver_movalue));

delete from hbrserverentity where vsrv_port = 443 AND NOT EXISTS (select hbrserver_movalue from secondarygroupentity where hbrserverentity.movalue = secondarygroupentity.hbrserver_movalue);

Start the HMS service: systemctl restart hms