VPXD on vCenter uses excess memory after a very long HTTP response.

Products

• VMware vCenter Server

Issue/Introduction

• There are instances when the VPXD service in vCenter after serving large responses is observed to increase its memory usage. It does not get back to normal unless the service is restarted.
  
  
• In case of vCenter 8.0 you may observer vCenter (vpxd) service crash cause by Out of Memory conditions.

Environment

VMware vCenter Server 7.0.3
VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Cause

There is a known issue with the internal HTTP/2 implementation, which is used by some of the vCenter services (most notably - by VPXD). Whenever some response has to be served it's being buffered in a data structure that can expand in size to accommodate responses of arbitrary size. Unfortunately, this data structure never shrinks. HTTP/2 sessions are reused, which causes these inflated buffers to retain their size forever.

For vCenter 8.0 some external solutions like backup can use a legacy API (PropertyCollector.retrieveProperties) to request extensive data properties. This legacy method forces the vCenter to generate a single, massive response in memory, causing service panic and unexpected termination.

Resolution

The issue will be resolved in future releases.

Workaround:

There are multiple workarounds to this issue. The user must try to apply them one after the other until the issue is resolved.
 

1. Decrease the number of pooled streams.

• A new configuration option is added under the name vmacore/http/http2/maxPooledStreams in /etc/vmware-vpx/vpxd.cfg. Setting this option to a smaller value than the default (1000) should decrease memory usage.

• Smaller values will decrease memory usage further, but will hinder performance more. This can be done,

a. Through advanced options in the UI

                Note: In the UI the option is called "config.vmacore.http.http2.maxPooledStreams"

b. Through command-line

  python /usr/lib/vmware-vpx/py/xmlcfg.py -f /etc/vmware-vpx/vpxd.cfg set vmacore/http/http2/maxPooledStreams 10 service-control --restart vpxd
 

    Note: The above options requires restarting VPXD to apply the changes.

 

     2.Increase VPXD memory (if possible)

          Manually increasing the heap memory on vCenter Server components in vCenter 6.x / 7.x
 

    3. Disable HTTP/2 :

This will cause VPXD to fall back on HTTP/1 for communication, which doesn't suffer from the memory issue, however this significantly increases file descriptor usage.

           It is achieved by removing the "http2:"
           prefix from all offending entries in the rhttpproxy config at: /etc/vmware-rhttpproxy/endpoints.conf.d/vpxd-rhttpproxy-endpoint.conf

          Use below command instead of deleting the entries manually to reduce errors:

     sed -i "s/http2://g" /etc/vmware-rhttpproxy/endpoints.conf.d/vpxd-rhttpproxy-endpoint.conf

     service-control --restart rhttpproxy