Segment allocation process error "too many connections for database <database_name>"

Products

• VMware Tanzu Greenplum

Issue/Introduction

Symptoms:

Error: Unexpected internal error (cdbgang.c:1447)

During the specified period of time, the master log for the database indicates that the error was generated during the allocation of the gang (as per the stack trace below):

2014-09-15 10:24:05.192697 IST,"xxxx_user","xxxdatabase",p14391,th1003468544,"172.28.250.0","29382",2014-09-15 10:24:0
3 IST,12646033,con82704,cmd4,seg-1,,dx205280,x12646033,sx1,"ERROR","XX000","Unexpected internal error (cdbgang.c:1447)
",,,,,,"--SO1_Queries
.........
..........
        select  XXXXXXXXXXXXXXXXX from XXXXXXXXXXX
 ......
 ......
         )
*/",0,,"cdbgang.c",1447,"Stack trace:
1    0xa6fdf9 postgres  (elog.c:468)
2    0xa74202 postgres elog_internalerror (elog.c:279)
3    0xb9bb4a postgres allocateGang (cdbgang.c:1519)
4    0x705c6d postgres AssignGangs (execUtils.c:1691)
5    0x6ebceb postgres ExecutorStart (execMain.c:549)
6    0x915ff9 postgres PortalStart (pquery.c:873)
7    0x90c704 postgres  (postgres.c:2451)
8    0x91067d postgres PostgresMain (postgres.c:4928)
9    0x876181 postgres  (postmaster.c:6801)
10   0x87c2c0 postgres PostmasterMain (postmaster.c:2346)
11   0x7811ba postgres main (main.c:212)
12   0x343ee1ecdd libc.so.6 __libc_start_main (??:0)
13   0x47cae9 postgres  (??:0)
"

The message below is produced by picking or checking any one of the segment logs during that period of time:

2014-09-15 10:24:04.858893 IST,"xxxx_user","xxxdatabase",p22043,th1196578560,"172.28.1.0","5919",2014-09-15 10:24:04 IST,8751573,con82704,,seg5,,,x8751573,sx1,"FATAL","53300","too many connections for database ""xxxdatabase""",,,,,,,0,,"postinit.c",238,
2014-09-15 10:24:04.964784 IST,"xxxx_user","xxxdatabase",p22067,th1196578560,"172.28.1.0","5973",2014-09-15 10:24:04 IST,8751575,con82704,,seg5,,,x8751575,sx1,"FATAL","53300","too many connections for database ""xxxdatabase""",,,,,,,0,,"postinit.c",238,

Environment

Cause

The issue affects non-superusers due to the limit of the connections that can be connected to the database. 

Querying the pg_database reveals that the database is limited to 50. As a result, when there are more than 50 connections querying the database, the database is unable to allocate gangs to address the new queries:

test=# select * from pg_database where datname='xxxdatabase';
   datname   | datdba | encoding | datistemplate | datallowconn | datconnlimit | datlastsysoid | datfrozenxid | dattablespace | datconfig |
        datacl
-------------+--------+----------+---------------+--------------+--------------+---------------+--------------+---------------+-----------+-----------------------------
----------------------------------------------------
 xxxdatabase     |     10 |        6 | f             | t            |           50 |         10899 |          803 |          1663 |           | {=Tc/gpadmin,gpadmin=CTc/gpa
dmin,xxxxdba=CTc/gpadmin,xxxx_user=CTc/gpadmin}
Time: 2.371 ms

Resolution

Increase the limit of database connections:

alter database <database_name> with connection limit <value>;

An alternate approach is to set the value to unlimited using:

alter database <database_name> with connection limit -1;