Aria Automation Config Job Submissions Fail with error "asyncpg.exceptions.TooManyConnectionsError: sorry, too many clients already"

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Sometimes when submitting a job targeting a large number of minions, many of the jobs fail resulting in the below error message being repeated in the RAAS logs (/var/log/raas/raas.log).

asyncpg.exceptions.TooManyConnectionsError: sorry, too many clients already

• Errors like these will be found in the postgres logs:

FATAL:  sorry, too many clients already

Environment

Aria Automation Config 8.x

Cause

The out-of-the-box installation of Postgresql has minimal settings defined in postgresql.conf

Resolution

1. Check the following values in the postgresql.conf file on the postgres SQL server to make sure the values are no less than the below values. 
   These are based on an example 4-8 core server with 16 GB of RAM and assume that Postgresql is the only service consuming the bulk of resources on the machine.
   
   Example path to file: /var/lib/postgresql/data/postgresql.conf
  
max_connections = 300
shared_buffers = 4096MB
work_mem = 16MB
max_wal_size = 4GB
checkpoint_completion_target = 0.9
effective_cache_size = 12288MB
   
   
2. After validating/modifying the values, restart the postgres service.
   systemctl restart postgresql

Additional Information

Tuning your PostgreSQL Server for Aria Automation Config