ESXi Hosts Unresponsive Due to Envoy Connection Limit Exhaustion by HCX-IX Appliances

Products

• VMware HCX

Issue/Introduction

• ESXi hosts in a cluster become unresponsive or disconnect from vCenter Server.

• The /var/log/envoy.log file on the affected ESXi host shows errors indicating that the maximum allowed HTTPS connections have been exceeded:

  <timestamps>180Z In(166) envoy[2102859]: "<timestamps>338Z warning envoy[21##45] [Originator@6876 sub=filter] [Tags: "ConnectionId":"31##76"] remote https connections exceed max allowed: 128"

• Below command will show total number of connections to ESXi envoy port 443 :
  Take SSH to the host which encounter the symptom : 
  esxcli network ip connection list | grep -i envoy | grep 443 -c

Environment

VMware HCX [ 20 or more Service Mesh with IX deployed ]
VMware ESXi

Cause

The ESXi envoy service has a hard limit of 128 remote HTTPS connections. VMware HCX Interconnect (IX) appliances maintain multiple connections to the ESXi hosts for migration operations (specifically, each IX appliance's internal hbrsrv process maintains 3 persistent port 443 connections).
When too many IX Appliances are deployed within a single service cluster, the aggregate connections exhaust the host's 128 connection threshold.

In observed instances, HCX-IX appliances were found to be using ~60 or more connections to a single ESXi host, directly leading to the host becoming unresponsive.

Resolution

To resolve and prevent connection limit exhaustion, you must restrict the number of IX Appliances per service cluster so that total concurrent https connections to ESXi remain well below the 128 limit per host.

Immediate Action Plan:

1. Remove Surplus Appliances: Identify and remove any surplus IX Appliances or Service Meshes (SMs) from the environment.

2. Preserve Network Extensions: If Network Extension (NE) Appliances are active within a Service Mesh alongside an IX appliance, remove only the IX Appliance from that specific SM. This ensures that necessary network extensions remain active and undisrupted.

Example Scenario: If you have an active scope of 400 workload VMs remaining to be migrated, approximately 10 IX Appliances within that given service cluster are more than sufficient based on Compute Profile configurations. Any IX appliances beyond this should be removed to maintain compliance with connection limits.