vCenter Server Appliance fails with the error: FATAL: remaining connection slots are reserved for non-replication superuser connections

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• vCenter Server is inacessible through the client.
• The VMware VirtualCenter Server service fails every few hours
• In the vpxd.log file, you see entries similar to:
  
  2012-11-05T08:35:54.929Z [7F4044195720 error 'Default'] A database error occurred: ODBC error: (08001) - [unixODBC]FATAL: remaining connection slots are reserved for non-replication superuser connections
2012-11-05T08:35:54.929Z [7F4044195720 error 'vpxdvpxdMain'] [Init] Init failed: VpxdVdb::Init(VpxdVdb::GetVcVdbInstId(), false, false, NULL)
Backtrace:
backtrace[00] rip 00007f403eefba84 Vmacore::System::Stacktrace::CaptureWork(unsigned int)
backtrace[01] rip 00007f403eddaae2 Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
backtrace[02] rip 00007f403ed3bda5 Vmacore::Throwable::Throwable(std::string const&)
backtrace[03] rip 00007f40450f727f Vmomi::Fault::SystemError::Exception::Exception(std::string const&)
backtrace[04] rip 00007f40450e035f /usr/lib/vmware-vpx/vpxd(+0xdcc35f) [0x7f40450e035f]
backtrace[05] rip 00007f40450f5ca3 /usr/lib/vmware-vpx/vpxd(+0xde1ca3) [0x7f40450f5ca3]
backtrace[06] rip 00007f40450d8c03 /usr/lib/vmware-vpx/vpxd(+0xdc4c03) [0x7f40450d8c03]
backtrace[07] rip 00007f403ce73c36 /lib64/libc.so.6(__libc_start_main+0xe6) [0x7f403ce73c36]
backtrace[08] rip 00007f40450d7fe9 /usr/lib/vmware-vpx/vpxd(+0xdc3fe9) [0x7f40450d7fe9]
2012-11-05T08:35:54.931Z [7F4044195720 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down...
   
• In the /storage/db/vpostgres/pg_log/ postgresql.log file, you see entries similar to:
  
  1 tm:2012-11-05 08:35:54.412 UTC db:VCDB pid:32351 FATAL: remaining connection slots are reserved for non-replication superuser connections
1 tm:2012-11-05 08:35:54.928 UTC db:VCDB pid:32376 FATAL: remaining connection slots are reserved for non-replication superuser connections
1 tm:2012-11-05 08:35:55.005 UTC db:VCDB pid:32383 FATAL: remaining connection slots are reserved for non-replication superuser connections

Environment

VMware vCenter Server 6.5.x
VMware vCenter Server 7.0.x
VMware vCenter Server Appliance 6.0.x
VMware vCenter Server Appliance 5.5.x
VMware vCenter Server 6.7.x
VMware vCenter Server Appliance 5.1.x

Cause

The vCenter Server Appliance uses vPostgres as its embedded database. The Windows version of vCenter Server can also be configured to use an embedded vPostgres database. By default, this caters to a maximum of 100 connections. This issue occurs when all available connection slots are in use.

Resolution

To resolve this issue:

vCenter Appliance:

1. Log in to the vCenter Server Appliance as the root user. The default password is vmware.
2. Back up the postgresql.conf file by running this command:
   
   # cp /storage/db/vpostgres/postgresql.conf /storage/db/vpostgres/postgresql.conf.orig
    
3. Open the postgresql.conf file using a text editor.
4. Find the max_connections entry in the file and set its value to 250:
   
   max_connections = 250
    
5. Save and close the file.
6. Reboot the vCenter Server Appliance for the changes to take effect.

Windows vCenter Server:

1. Connect to the vCenter server via Remote Desktop Connection or console
2. Log into the server as an administrator user.
3. Open a file manager window and find the configuration file located at:
   
   %ProgramData%\VMware\vCenterSErver\data\vpostgres\
4. Open the postgresql.conf file using a text editor.
5. Find the max_connections entry in the file and set its value to 250:
   
   max_connections = 250
    
6. Save and close the file.
7. Reboot the Windows vCenter Server for the changes to take effect.

Note: Increasing max_connections uses approximately 400 bytes of shared memory per connection slot in addition to the lock space. Each lock table slot uses approximately 270 bytes of shared memory, and there are max_locks_per_transaction * (max_connections + max_prepared_transactions) lock table slots.