Managing Idle connection timeout for the Envoy on the ESXi Servers

Products

• VMware vSphere ESXi

Issue/Introduction

• The ESXi Server utilizes the Envoy proxy to manage inbound connections on port 443, with a default limit of 128 simultaneous connections.

• When this connection limit is reached, often due to numerous "half-open" connections resulting from clients abruptly closing their sessions, the vCenter Server may fail to connect to the ESXi Server. These half-open connections are eventually closed and cleaned up by Envoy upon timeout.

• However, the default timeout period is excessively long, causing a significant delay in cleanup. This prolonged delay leaves the ESXi Server inaccessible for an extended duration.

This article outlines the procedure for modifying the idle timeout configuration for the Envoy proxy on the ESXi Server to address this issue.

 

Resolution

Prerequisite:

Update the ESXi Server to ESXi 8.0 Update 3h - Build 25067014 where this feature is introduced.

Note: Though the configuration is related to Envoy's handling of idle connections, setting is applied to rhttpptoxy service as it is used to pass the configuration settings to Envoy. 

Using Configuration Profile:

• To modify the idle connection timeout for ESXi Servers that are part of a cluster with Configuration Profiles enabled, update the Desired state Configuration for the "rhttpproxy" service.
• The specific setting is named "idle_connection_timeout" within the "envoy" group. This value is specified in seconds.
• Recommendation:
  • Set this timeout to align with the vCenter Server's idle connection timeout (which defaults to 900 seconds).
  • For a safer implementation, it is advisable to set the ESXi value to be 2 seconds greater than the vCenter Server's value to prevent a race condition in connection closure.
• Once updated, you can remediate the host to apply the new changes. Changes will be effective post the server is restarted.

Reference:

Updating configuration directly on the ESXi Server:

Another approach is to update the settings directly on each ESXi Server and restart the rhttpproxy service.

Steps:

• Login to ESXi Server over SSH as root user
• Extract the default configuration for service rhttpproxy
/bin/configstorecli config default get -c esx -g services -k rhttpproxy -outfile tmp.json
• Edit the output file "tmp.json" to include the settings.
  • New configuration parameter named "idle_conenction_timeout" with value in seconds need to be added under the "envoy" block (Ex: "idle_connection_timeout": 902,)
  • Before change
    
  • After change
    
  • Save the file
• Apply the updated service configuration
  /bin/configstorecli config current set -c esx -g services -k rhttpproxy -infile tmp.json
  
  • Results of this action will be updated to the console ("Set: completed successfully" suggests changes are applied)
• Restart the rhttpproxy service
   /etc/init.d/rhttpproxy restart
• Once this changes is done and rhttpproxy service is restarted, all existing and future connections will be managed with this new settings. 

Additional Information

Reference: ESXi host is unresponsive in vCenter because the maximum number of allowed HTTPS connections has been exceeded.