ESXI hosts go unresponding on the vCenter Server with Envoy proxy maximum connection limit exceeded - vSphere Replication

Products

• VMware Live Recovery
• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

• ESXi hosts randomly go unresponding on the vCenter server
• This happens after VMs are configured with enhanced replication
• The CPU utilization of the VRMS servers may go up to 100% when the issue occurs
• In /var/run/log/envoy.log you will notice the below warnings:

YYYY-MM-DD-HH-MM-SS In(166) envoy[2099870]: "YYYY-MM-DD-HH-MM-SS warning envoy[2100520] [Originator@6876 sub=filter] [Tags: "ConnectionId":"2015172"] remote https connections exceed max allowed: 128"

• The maximum connection limit is reached when vSphere Replication does not close HTTP connections which are created as part of the health checks during the configuration of enhanced replication:

Proto  Recv Q  Send Q  Local Address        Foreign Address      State        World ID  CC Algo  World Name
-----  ------  ------  -------------------  -------------------  -----------  --------  -------  ----------
tcp         0       0  xx.xx.xx.xx:443    xx.xx.xx.xx:54108   ESTABLISHED  35101291  newreno  envoy
tcp         0       0  xx.xx.xx.xx:443    xx.xx.xx.xx:54100   ESTABLISHED  35101291  newreno  envoy

• In /var/run/log/envoy-access.log you will notice the below connections open for hours:

YYYY-MM-DD-HH-MM-SS In(166) envoy-access[2099882]: GET /hbragent/api/v1.0/appPing?broker_ip=xx.xx.xx.xx&broker_port=32032&group=PING-GID-5243c529-e210-xxxx 200 via_upstream - 0 387 - 107 106 0 xx.xx.xx.xx:34164 HTTP/1.1 TLSv1.2 xx.xx.xx.xx:443 - HTTP/1.1 - /var/run/vmware/hbragent-rest-tunnel - -

Environment

• ESXi 8.x
• vSphere Replication 8.8 / 9.x
• vSphere Live Site Recovery 9.x
• vCenter 8.x / 9.x

Cause

The Envoy proxy service reaches the maximum concurrent connection limit (128). This occurs because vSphere Replication does not properly close HTTP connections created during health checks for enhanced replication.

Resolution

This issue is resolved in vSphere Replication 9.0.2.3. To download vSphere Replication 9.0.2.3 go to Broadcom Support Portal

Workaround:

1. Open a SSH session to the VRMS server on both the sites.
2. Open file /opt/vmware/hms/conf/hms-configuration.xml with a text editor
3. Set schedule-health-checks to false
4. Restart HMS service on both sites

systemctl restart hms

     5. When configuring enhanced replication, click "Next" to skip the health check.

Additional Information

Note: For defects and enhancements, we recommend subscribing to this article at https://knowledge.broadcom.com/external/article/275360 to receive updates on fix status.

vSphere Replication 9.0.2.3 Release Notes