ESXi Hosts Show as "Not Responding" Due to Envoy Session Limits Exceeded by Replication Services

Products

• VMware vCenter Server

Issue/Introduction

ESXi hosts intermittently appear as "Not Responding" in vCenter Server while VMs remain operational. The vCenter web interface may become unresponsive or display HTTP 401 errors. Users accessing VMs through vCenter lose connectivity during these periods.

Environment

• VMware ESXi 7.0.x and newer
• VMware vCenter Server 7.0.x and newer
• Environment using replication services (such as Veeam Replication or vSphere Replication)
• ESXi hosts show as "Not Responding" in vCenter Server
• Host https:// management interface becomes unreachable
• VMs continue running without interruption
• Host remains pingable during outages
• Issue may affect multiple hosts in sequence
• Condition resolves temporarily after host reboot

Cause

Replication services can create more HTTPS sessions than it closes, and more than the ESXi host's envoy service can handle. The envoy service has a limit of 128 concurrent HTTPS sessions. When this limit is exceeded, connection failures occur between vCenter and the host.

Evidence in host envoy logs:

warning envoy[2100277] [Originator@6876 sub=filter] [Tags: "ConnectionId":"12521751"] remote https connections exceed max allowed: 128

Resolution

Immediate Work-around

1. Identify the replication service creating excessive connections by checking envoy.log for the source IP
2. Temporarily disable the identified replication service
3. Restart the envoy service on the affected host:

      /etc/init.d/envoy restart

Long-term Solution

1. Update replication software to latest version
2. Configure replication jobs to limit concurrent sessions
3. If issues persist:
   • Implement firewall rules to limit concurrent connections from replication servers
   • Contact replication software vendor for additional guidance

Additional Information

• This issue can occur with any add-on solution that creates multiple concurrent HTTPS connections to ESXi hosts
• The envoy service manages HTTPS connections between vCenter Server and ESXi hosts
• VM operations remain unaffected as they don't rely on these management connections
• vSphere Replication sending large number of HTTPS requests to envoy proxy causing hostd to crash - remote https connections exceed max allowed