Greenplum fails to start with error "could not create semaphores: No space left on device"
Article ID: 297014
Updated On:
Products
Issue/Introduction
Greenplum fails to start with error "could not create semaphores: No space left on device"
Segment log has the following error:
2020-05-25 11:10:15.314132 UTC,,,p43438,th-1103329408,,,,0,,,seg-1,,,,,"FATAL","XX000","could not create semaphores: No space left on device (pg_sema.c:129)","Failed system call was semget(6001001, 17, 03600).","This error does *not* mean that you have run out of disk space. It occurs when either the system limit for the maximum number of semaphore sets (SEMMNI), or the system wide maximum number of semaphores (SEMMNS), would be exceeded. You need to raise the respective kernel parameter. Alternatively, reduce PostgreSQL's consumption of semaphores by reducing its max_connections parameter (currently 753). The PostgreSQL documentation contains more information about configuring your system for PostgreSQL.",,,,,,"InternalIpcSemaphoreCreate","pg_sema.c",129,1 0x96262b postgres errstart (elog.c:521) 2 0x7c7c45 postgres PGSemaphoreCreate (pg_sema.c:114) 3 0x829fcf postgres InitProcGlobal (discriminator 3) 4 0x81f1ac postgres CreateSharedMemoryAndSemaphores (ipci.c:338) 5 0x7e077c postgres PostmasterMain (postmaster.c:4005) 6 0x4cb867 postgres main (main.c:206) 7 0x7f8cb98fb505 libc.so.6 __libc_start_main + 0xf5 8 0x4cbe1c postgres <symbol not found> + 0x4cbe1c
Cause
If this is a Redhat OS, a RHEL limit may have ben reached. See On RHEL servers, changing the semaphore value fails with a message "setting key "kernel.sem": Numerical result out of range".
If the SEMMNI value in kernel.sem is set higher than 32768 it will be considered out of range and the OS will use the default value which is 128.
The following message can be seen in the /var/log/messages log file.
systemd-sysctl[366]: Failed to write '500 2048000 200 40960' to '/proc/sys/kernel/sem': Numerical result out of range
The default value of 128 is too low for Greenplum and will not allow Greenplum to start.
To check the current kernel setting:
$ sysctl kernel.sem kernel.sem = 500 2048000 200 40960
Refer to Configuring Your Systems in the Installation Guide for the version of Greenplum installed on the cluster.
Note: The same limit of 32768 also applies to kernel.msgmni and kernel.shmmni
Resolution
Fix
As root user, use sysctl command to change the value to the recommended value. For example:
sysctl -w kernel.sem="250 2048000 200 8196"
Note: Refer to Configuring Your Systems for the correct values for the particular version of Greenplum
This needs to be set on all hosts in the cluster.
The setting change above will not survive a host reboot. To ensure the setting persists after a reboot:
As root user change the value of kernel.sem in /etc/sysctl.conf on all hosts in the cluster.
See the following RedHat documentation for further details on the kernel setting:
8.5. Setting Semaphore Parameters
8.6. An Example of Semaphore Settings
On RHEL servers, changing the semaphore value fails with a message "setting key "kernel.sem": Numerical result out of range".
Feedback
