VCFA UI is inaccessible with "No healthy upstream" error due to exhausted database connections

Products

• VCF Automation

Issue/Introduction

Users are unable to access the VCF Automation UI, which displays a "No healthy upstream" error.

Tenant-manager pods are in Crashloopback because they cannot connect to the database.

This can be verified by checking the logs under: /var/log/services-logs/prelude/tenant-manager/file-logs/tenant-manager-app.log

Fatal exception caught during application bootstrap. Halting application. Message: 'Error creating bean with name 'globalDataSourceService' defined in class path resource [com/vmware/vcloud/common/main/bootstrap/phase2/bootstrap-datasource-context.xml]: Cannot resolve reference to bean 'globalDataSource' while setting bean property 'targetService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'globalDataSource': FactoryBean threw exception on object creation; nested exception is org.postgresql.util.PSQLException: FATAL: remaining connection slots are reserved for non-replication superuser connections'

/var/log/services-logs/prelude/vcfapostgres-0/file-logs/vcfapostgres-0.log

dataprotection_server_db dataprotection_server_db_owner_user  2LOG:  connection authenticated: identity="dataprotection_server_db_owner_user" method=scram-sha-256 (/home/postgres/pgdata/pgroot/data/pg_hba.conf:12)
dataprotection_server_db dataprotection_server_db_owner_user  3LOG:  connection authorized: user=dataprotection_server_db_owner_user database=dataprotection_server_db SSL enabled (protocol=TLSv1.3, bits=256)
dataprotection_server_db dataprotection_server_db_owner_user 4FATAL:  remaining connection slots are reserved for non-replication superuser connections

Environment

VCF Automation 9.0.x

Cause

When snapshot-based backups are requested for the VCFA VMs, it completely consumes the available Postgres database connections. Because the connection pool is exhausted, the tenant-manager fails to connect to the database and generates a fatal error: "FATAL: remaining connection slots are reserved for non-replication superuser connections."

Resolution

To resolve this issue, perform the following steps:

1. Exclude the VCFA VMs from your snapshot-based backup schedules as snapshot is not supported for VCFA VM's 

2. Reboot the VCFA nodes to clear the exhausted connection slots and restore normal operations.