vSphere Replication Management Server (VRMS) partition /opt/vmware/heapdump fills to 100% due to Envoy connection limits

Products

• VMware vSphere ESXi

Issue/Introduction

The vSphere Replication Management Server (VRMS) appliance experiences critical disk space exhaustion on the /opt/vmware/heapdump partition.

• The VRMS service may become unstable or unresponsive.

• Resizing the disk via LVM provides only temporary relief; the partition fills again rapidly (often within 24 hours).

Symptoms in VRMS logs (hbrsrv.log or hms.log):

error hbrsrv[PID] [Originator@6876 sub=IO.Http opID=...] User agent failed to send request; (null), N7Vmacore15SystemExceptionE(stream truncated: The connection was closed by the remote end during handshake.

Symptoms in ESXi Host logs (/var/log/vmware/envoy/envoy.log):

warning envoy[PID] [Originator@6876 sub=filter] ... remote https connections exceed max allowed: 128
warning envoy[PID] [Originator@6876 sub=filter] ... closing connection TCP<VRMS_IP:PORT, ESXI_IP:443>

Environment

VMware vSphere Replication 8.x / 9.x
Large environments (e.g., >500 hosts)

 

Cause

The VRMS appliance is establishing a high volume of concurrent HTTPS connections to ESXi hosts, exceeding the hard limit of 128 connections enforced by the ESXi sidecar proxy (envoy).

When this limit is reached, the ESXi host forcibly closes additional connections during the handshake. This results in stream truncated exceptions on the VRMS side, causing service instability and the generation of massive heap dumps that rapidly consume the /opt/vmware/heapdump partition.

Resolution

This issue is resolved in vSphere Replication 9.0.2.3

ESXi hosts are intermittently disconnected from the vCenter but automatically reconnects after a few seconds or minutes.

If the ESXi envoy reaches the limit of 128 concurrent HTTPS sessions, most of which are from vSphere Replication, you observer the follwoing error.

2025-06-26T06:59:55.859Z In(166) envoy[2099870]: "2025-06-26T06:59:46.278Z warning envoy[2100520] [Originator@6876 sub=filter] [Tags: "ConnectionId":"2015172"] remote https connections exceed max allowed: 128"