Out of Memory (OOM) in VMware Cloud Director 10.6.x due to API-driven Database Connection Pooling

Products

• VMware Cloud Director

Issue/Introduction

• VMware Cloud Director (VCD) cells experience high memory utilization or enter an Out of Memory (OOM) state.
• High frequency of API calls is observed in the environment (Extra Large deployment profile).
• The primary cell may become unresponsive, or the vmware-vcd service restarts unexpectedly.
• Database connection counts remain high even during periods of lower activity.

Environment

VMware Cloud Director 10.6.1.1

Cause

When Cloud Director 10.6.x faces high API pressure, the default database settings can leave unused connections open unnecessarily.

Resolution

Perform the following steps on all VMware Cloud Director cells in the server group:

1. Establish an SSH session to the VCD cell as root.
2. Backup the configuration file before modification:
   
   cp /opt/vmware/vcloud-director/etc/global.properties /opt/vmware/vcloud-director/etc/global.properties.bak
   
   
3. Edit the file /opt/vmware/vcloud-director/etc/global.properties using a text editor (e.g., vi or nano).
4. Append the following parameters to the end of the file:
   
   database.pool.minEvictableIdleTimeMillis=6000
database.pool.timeBetweenEvictionRunsMillis=30000 
   
   Note: minEvictableIdleTimeMillis sets the minimum time an object may sit idle in the pool before it is eligible for eviction, timeBetweenEvictionRunsMillis sets the interval between eviction thread runs.
   
   
5. Save and close the file.
6. Restart the VCD service to apply changes:
   
   systemctl restart vmware-vcd
   
   
7. Monitor the cell stability. Use top or vcloud-container-debug.log to verify that memory utilization stabilizes and idle database connections are properly cleaned up.