Segment will not start with error "Failed system call was semget" in VMware Tanzu Greenplum

Products

• VMware Tanzu Greenplum

Issue/Introduction

Cannot start Greenplum Database (GPDB) due to the following error:

20200515:08:28:20:011494 gpstart:mdw:gpadmin-[INFO]:-DBID:18 FAILED host:'sdw2' datadir:'/data1/gp6/mirror/gpseg0' with reason:'PG_CTL failed.'

The segment startup.log file reports:

2020-05-15 08:28:08.338759 CEST,,,p2697,th1511610496,,,,0,,,seg0,,,,,"FATAL","XX000","could not create semaphores: No space left on device (pg_sema.c:126)","Failed system call was semget(11000001, 17, 03600).","This error does *not* mean that you have run out of disk space. It occurs when either the system limit for the maximum number of semaphore sets (SEMMNI), or the system wide maximum number of semaphores (SEMMNS), would be exceeded. You need to raise the respective kernel parameter. Alternatively, reduce PostgreSQL's consumption of semaphores by reducing its max_connections parameter. The PostgreSQL documentation contains more information about configuring your system for PostgreSQL.",,,,,,"InternalIpcSemaphoreCreate","pg_sema.c",126,
1 0x9f392c postgres errstart (elog.c:558)
2 0x801648 postgres PGSemaphoreCreate (pg_sema.c:113)
3 0x8775ba postgres InitProcGlobal (proc.c:258)
4 0x863825 postgres CreateSharedMemoryAndSemaphores (ipci.c:288)
5 0x815ccf postgres PostmasterMain (postmaster.c:1334)
6 0x4ceb31 postgres main (main.c:205)
7 0x7fd055b00555 libc.so.6 __libc_start_main + 0xf5
8 0x4cf10c postgres + 0x4cf10c

Environment

Product Version: 6.3

Resolution

Check the kernel parameter kernel.sem on the hosts in the cluster.

gpssh -f <hostfile> sysctl kernel.sem

This should report values similar to:

kernel.sem = 500 2048000 200 4096

Note: It important to check the recommended values for the version of GPDB running on the hosts in the cluster. The recommended value may vary between versions. The recommended values can be found in the VMware Tanzu Greenplum Installation Guide.

For information on the parameter and the individual values see Setting Semaphore Parameters and Setting Semaphores.