HTTPS Connection Exhaustion in envoy logs of the vCenter server
search cancel

HTTPS Connection Exhaustion in envoy logs of the vCenter server

book

Article ID: 344920

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

  • Unable to load the vCenter Server UI page; it displays "This site can't be reached"

  • It may also show "Secure Connection Failed – An error occurred while connecting to the vCenter FQDN: PR_END_OF_FILE_ERROR"

  • Envoy logging var/log/vmware/envoy/envoy.log in the vCenter server has the following snippets,

     warning envoy[40146] [Originator@6876 sub=filter] [C15632615] remote https connections exceed > max allowed: 2048
 warning envoy[40146] [Originator@6876 sub=filter] [C15632615] closing connection TCP<##.###.##.##.###:41704, ##.###.##.##.###:443>

  • In certain circumstances, remote https connections to vCenter may become exhausted.

  • By default, these connections are set to a maximum of 2048 connections.

  • One possible contributor to the limit being reached:

    • When using VDDK 7.0.0 in a backup solution, it increases the amount of connections to vCenter's Envoy proxy service. When this limit is reached, it may result in failed backups as part of those backup workflows.

  • To confirm that the limit is being reached, please do the following:

    • VDDK logging (file path depends on backup software):

       error -> [297506] [Originator@6876 sub=IO] HandshakeCb; <SSL(<io_obj p:0x00007fb7d806d1b8, h:32, <TCP '##.###.##.##.###:41704'>, <TCP '##.###.##.##.###: 443'>>)>; error: N7Vmacore15SystemExceptionE(Connection reset by peer: The connection is terminated by the remote end with a reset packet. Usually, this is a sign of a network problem, timeout, or service overload.)

Note:

This article does NOT apply to local http/https connections happening within the vCenter Appliance such as service to service connections over the local loopback interface. (e.g. vsan-health making calls to VUM), it provides a way to remediate the VC connection limit problem.

Environment

VMware vCenter Server 7.0.x

VMware vCenter Server 8.0.2

VMware Virtual Disk Development Kit 7.0.0

Cause

  1. The limit that is more easily reached relates to vCenter Envoy proxy service since vCenter Server 7.0.x

  2. When using VDDK 7.0.0 in a backup solution, it increases the amount of connections to the Envoy proxy service, increasing the likelihood that the maxRemoteHttpsConnection limit is reached. When this limit is reached, it may result in failed backups as part of those backup workflows. 

Resolution

Workaround:

    • To avoid or remediate this scenario, a number of workarounds exist which can be applied in isolation or in parallel as needed:

    • Upgrade to 7.0 U1 or later VDDK version to avail of code improvements  

To Increase the maximum RemoteHttpsConnections limit. See following steps:

    • Change envoy's proxy limit for external inbound HTTPS connections, the following changes are needed in the config options according to vCenter version:

    • For vCenter Server 7.0.2 and later, change/add this config option “maxRemoteHttpsConnections” in /etc/vmware-rhttpproxy/config.xml

    • SSH to the vCenter server and take a backup of the config.xml file: cp /etc/vmware-rhttpproxy/config.xml /etc/vmware-rhttpproxy/config.xml.bkp

    • Using vi editor open the config.xml file and add the maxRemoteHttpsConnections under envoy tag:

<config>
   <envoy>
      <L4Filter>
         <maxRemoteHttpsConnections>3072</maxRemoteHttpsConnections>
      </L4Filter> 
   </envoy>
</config>

    • For vCenter Server 7.0.1 (Recommended minimum version) , change/add the config options “maxHttpConnections” and “maxHttpsConnections” in /etc/vmware-rhttpproxy/config.xml 
    • For vCenter Server 7.0.0, change the config option “max_http_connections” and “max_https_connections” in /etc/vmware-envoy/config.yaml

 

Note: vCenter Server 7.x has multiline comments in the /etc/vmware-rhttpproxy/config.xml file so there is a need to uncomment the desired option when change of the value is needed

Example:

Before:

  <L4Filter>
      <!--
      <maxRemoteHttpsConnections>2048</maxRemoteHttpsConnections>

After: 

  <L4Filter>
      <maxRemoteHttpsConnections>3072</maxRemoteHttpsConnections>
      <!--

    • After the configuration has been updated and the file has been saved, reload the configuration by one of the two ways:

    1. Restart rhttpproxy service : service-control --restart rhttpproxy

    2. Send SIGHUP signal to rhttpproxy service: kill -1 `pidof rhttpproxy`

 

NOTE:

While no hard limit exists to the maximum configurable limit, it is recommended to increase with caution as an overly extended limit may impact VC performance due to excessive connections and subsequent resource consumption.

Increasing to 3072 is verified and suitable for most cases. If it is need to increase to number > 3072, please contact Broadcom support so that help can be provided to understand and analyze overly excessive connections in the environment.

Additional Information

From vSphere 8.0u3 (VCF 5.2) : vCenter Server Appliance introduces a shared connection pool for HTTP and HTTPS traffic. This enhancement allows unused HTTP connection slots to be dynamically utilized by HTTPS requests, improving flexibility and reducing connection failures.

Scaling vCenter Server Connections for Improved Resiliency

Powered by Wolken Software