High Memory Consumption in vStats and Envoy-HGW

Products

• VMware vSphere ESXi 8.0
• VMware vSphere ESXi

Issue/Introduction

Sometimes in the VC, we see that vStats service or envoy-hgw service is using 5+ GB (sometimes it can go much higher than this value) of memory. This may be caused due to the data being written to http buffer faster than it can be read. This is due to the change in the HTTP protocol version from HTTP1 to HTTP2.

We can verify the memory usage of the services using 
        top -p `pidof envoy-hgw` -b -n1  | tail -n1

Note: We need to look at the physical memory usage and not the virtual memory usage. The virtual memory can be higher and it is okay to be at those values.

Environment

VMware vCenter 9.0
ESXi 8.0 U2 onwards.
Customer is using vStats tech preview APIs for monitoring data collection.

Cause

On data collection enablement by customers using vStats tech preview APIs, vStats starts the data collection from configured ESXi hosts. In VC version 9.0, vStats service started using HTTP 2.0 streaming with ESXi hosts (>8.0U2 version) to collect the monitoring data. In previous releases and host versions, HTTP 1 PULL is used to collect the data. The default maximum buffer size for HTTP 2 connection is 256 MB. The HTTP 2 buffer can maximum grow till 256 MB unlike HTTP 1 where the limit was ~2-3 MB. Hence this might lead to high memory consumption in vStats and Envoy-hgw.

Resolution

Workaround 1

1. Restart vstats service -> This will kill all the HTTP 2 connection including potentially leaking streams.
2. If this does not solve the issue, stop the data collection enabled via tech preview APIs by deleting AcqSpecs. 
          

Workaround 2 

1. Update /etc/vmware-envoy-hgw/listeners.yaml
           http2_protocol_options:
          allow_connect: true
          hpack_table_size: 4096
          initial_connection_window_size: 2097152
          initial_stream_window_size: 262144
          max_concurrent_streams: 2147483647
2. Change values of initial_connection_window_size and initial_stream_window_size to above given values.
3. Restart vmware-envoy-hgw service.
4. Restart vstats service.