PostgreSQL Connections Exceeding Limit causing the portal to go down Due to Parallel Export Configuration Operations

Products

• VMware Avi Load Balancer

Issue/Introduction

PostgreSQL connections are exceeding the limit of 750 rapidly, leading to potential performance degradation. This issue occurs when multiple export configuration operations are performed in parallel, causing a connection leak due to orphaned subprocesses.

Environment

GCP

Cause

The issue arises when the maintenanceportal worker process spawns multiple subprocesses to handle export configurations in parallel. If the worker process is force-killed during this operation, its subprocesses become orphaned, leading to a connection leak where they hold onto database connections. This behavior is triggered under rare conditions but can occur if multiple export operations are run simultaneously.

 

Resolution

Workaround:

1: Reduce potential for connection leak
Modify clients so that they are not calling export configuration in parallel
In controller properties - set controller_resource_info_collection_period = 525600

Login to controller shell and execute the command below. 
> configure controller properties
controller_resource_info_collection_period 525600
save

2: Monitor setup - if connection leak still happens with the suggested modifications:
restart maintenanceportal using "avictl stop maintenanceprtal.service"

3: Security fix
use POST instead of GET for export configuration and move parameters into post data
Ensure passphrase is not included in the query parameters!

 

Permanent Fix:

The issue has been resolved in versions 30.2.1 and 31.1.1 of the controller. The fix includes replacing the Python export configuration routine with a Golang implementation, which offers improved process and connection management during parallel processing.
The patch is not available for 22.x versions.