vCenter 7.0.x HTTP Connection Exhaustion

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

This article provides a way to remediate the VC connection limit problem.

Note: This article does NOT apply to local http/https connections happening within the vCenter Appliance such as service to service connections over the local loopback interface. (e.g. vsan-health making calls to VUM)

Symptoms:

In certain circumstances, remote https connections to vCenter may become exhausted.
By default, these connections are set to a maximum of 2048 connections.
 
One possible contributor to the limit being reached:
When using VDDK 7.0.0 in a backup solution, it increases the amount of connections to vCenters Envoy proxy service. When this limit is reached, it may result in failed backups as part of those backup workflows.
 
To confirm that the limit is being reached, please do the following:
 
VDDK logging (file path depends on backup software):

2022-02-09T15:15:47.444+02:00 error -> [297506] [Originator@6876 sub=IO] HandshakeCb; <SSL(<io_obj p:0x00007fb7d806d1b8, h:32, <TCP '##.###.##.##.###:41704'>, <TCP '##.###.##.##.###: 443'>>)>; error: N7Vmacore15SystemExceptionE(Connection reset by peer: The connection is terminated by the remote end with a reset packet. Usually, this is a sign of a network problem, timeout, or service overload.)
 
VC Envoy logging (/var/log/vmware/envoy/envoy.log):

2022-02-09T13:15:47.443Z warning envoy[40146] [Originator@6876 sub=filter] [C15632615] remote https connections exceed > max allowed: 2048
2022-02-09T13:15:47.443Z warning envoy[40146] [Originator@6876 sub=filter] [C15632615] closing connection TCP<##.###.##.##.###:41704, ##.###.##.##.###:443>

 

Environment

VMware vCenter Server 8.0.x
VMware vCenter Server 7.0.x

Cause

1. The limit that is more easily reached relates to vCenter Envoy proxy service since vCenter 7.0.x.
2. When using VDDK 7.0.0 in a backup solution, it increases the amount of connections to the Envoy proxy service, increasing the likelihood that the maxRemoteHttpsConnection limit is reached. When this limit is reached, it may result in failed backups as part of those backup workflows. 

Resolution

None 

Workaround:

To avoid or remediate this scenario, a number of workarounds exist which can be applied in isolation or in parallel as needed:

Upgrade to 7.0 U1 or later VDDK version to avail of code improvements  

Increase the maximum RemoteHttpsConnections limit. See following steps:

 
To change envoy's proxy limit for external inbound HTTPS connections, the following changes are needed in the config options according to vCenter version:

For vCenter Server 7.0.2 and later,  change/add this config option “maxRemoteHttpsConnections” in /etc/vmware-rhttpproxy/config.xml 

<config>
   <envoy>
      <L4Filter>
         <maxRemoteHttpsConnections>3072</maxRemoteHttpsConnections>
      </L4Filter> 
   </envoy>
</config>

For vCenter Server 7.0.1(Recommended minimum version) ,   change/add the config options “maxHttpConnections” and “maxHttpsConnections” in /etc/vmware-rhttpproxy/config.xml 

For vCenter Server 7.0.0,  change the config option “max_http_connections” and “max_https_connections” in /etc/vmware-envoy/config.yaml

 

Note vCenter Server 7.x has multiline comments in the /etc/vmware-rhttpproxy/config.xml file so there is a need to uncomment the desired option when change of the value is needed e.g.

Before:

  <L4Filter>
      <!--
      <maxRemoteHttpsConnections>2048</maxRemoteHttpsConnections>

After: 

  <L4Filter>
      <maxRemoteHttpsConnections>3072</maxRemoteHttpsConnections>
      <!--

After the configuration has been updated and the file has been saved, reload the configuration by one of the two ways:
1). Restart rhttpproxy service:
$ service-control --restart rhttpproxy
2). Send SIGHUP signal to rhttpproxy service:
$ kill -1 `pidof rhttpproxy` 
 

NOTE: While no hard limit exists to the maximum configurable limit, it is recommended to increase with caution as an overly extended limit may impact VC performance due to excessive connections and subsequent resource consumption. Increasing to 3072 is verified and suitable for most cases. If it is need to increase to number > 3072, please contact VMware support so that help can be provided to understand and analyze overly excessive connections in the environment.