UI and CLI becomes unresponsive in Avi version 22.1.x due to export config failures

Products

• VMware Avi Load Balancer

Issue/Introduction

in 22.1.x, Customer may experience a scenario where they may lose the UI and CLI access to the controller, after they have attempted to do multiple export configuration simultaneously.

 

As seen below, multiple API call failures to /api/configuration/export may be seen resulting in the error "FATAL:  sorry, too many clients already"

/var/lib/avi/log/portal-webapp.log

[2025-05-27 19:00:13,520] ERROR [error_handler.process_exception:95] ^[[31mError in GET /api/configuration/export : ^[[0m
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/django/db/backends/base/base.py", line 130, in ensure_connection
    self.connect()
.
.
psycopg2.OperationalError: connection to server at "node2.controller.local" (127.0.0.1), port 5000 failed: FATAL:  sorry, too many clients already

Environment

Avi Loadbalncer version 22.1.x

Cause

In case of multiple internal failures in export configuration, a large number of idle connections can take up database connections which can accumulate over time and cause the maximum limit of database connections to get exhausted. 

Portal logs may be filled with below logs.

/var/log/upstart/aviportal.log


aviportal.log.4.gz:django.db.utils.OperationalError: connection to server at "node2.controller.local" (127.0.0.8), port 5000 failed: FATAL:  remaining connection slots are reserved for non-replication superuser connections
aviportal.log.4.gz:psycopg2.OperationalError: connection to server at "node2.controller.local" (127.0.0.8), port 5000 failed: FATAL:  remaining connection slots are reserved for non-replication superuser connections
aviportal.log.4.gz:django.db.utils.OperationalError: connection to server at "node2.controller.local" (127.0.0.8), port 5000 failed: FATAL:  remaining connection slots are reserved for non-replication superuser connections
aviportal.log.4.gz:psycopg2.OperationalError: connection to server at "node2.controller.local" (127.0.0.8), port 5000 failed: FATAL:  remaining connection slots are reserved for non-replication superuser connections
aviportal.log.4.gz:django.db.utils.OperationalError: connection to server at "node2.controller.local" (127.0.0.8), port 5000 failed: FATAL:  remaining connection slots are reserved for non-replication superuser connections

 

Note: This issue happens only in 22.1.x, when there are multiple parallel /api/configuration/export API call failures.

Resolution

Workaround: systemctl restart maintenanceportal.service or do a warm reboot of the leader node

 

Issue is not fixed in 22.1.x version and this is not applicable to 30.x or 31.x versions due to code change to golang.