gpinitsystem Error: "Failed to complete obtain psql count Master gp_segment_configuration"

Products

• VMware Tanzu Greenplum

Issue/Introduction

Symptoms:

When attempting a Greenplum system initialization, it failed with an error message,

“Failed to complete obtain psql count Master gp_segment_configuration.”

Error Message:

[gpadmin@gssapj-dev2 gp]$ gpinitsystem -a -c gpinitsystem_singlenode_5.0 --max_connections=30
20170419:01:10:33:116684 gpinitsystem:gssapj-dev2:gpadmin-[INFO]:-Checking configuration parameters, please wait...
20170419:01:10:33:116684 gpinitsystem:gssapj-dev2:gpadmin-[INFO]:-Reading Greenplum configuration file gpinitsystem_singlenode_5.0
20170419:01:10:33:116684 gpinitsystem:gssapj-dev2:gpadmin-[INFO]:-Locale has not been set in gpinitsystem_singlenode_5.0, will set to default value
20170419:01:10:33:116684 gpinitsystem:gssapj-dev2:gpadmin-[INFO]:-Locale set to en_US.utf8
20170419:01:10:33:116684 gpinitsystem:gssapj-dev2:gpadmin-[INFO]:-Detected a single host GPDB array build, reducing value of BATCH_DEFAULT from 60 to 4
20170419:01:10:33:116684 gpinitsystem:gssapj-dev2:gpadmin-[INFO]:-Checking configuration parameters, Completed
20170419:01:10:33:116684 gpinitsystem:gssapj-dev2:gpadmin-[INFO]:-Commencing multi-home checks, please wait...
.
20170419:01:10:34:116684 gpinitsystem:gssapj-dev2:gpadmin-[INFO]:-Configuring build for standard array
20170419:01:10:34:116684 gpinitsystem:gssapj-dev2:gpadmin-[INFO]:-Commencing multi-home checks, Completed
20170419:01:10:34:116684 gpinitsystem:gssapj-dev2:gpadmin-[INFO]:-Building primary segment instance array, please wait...
..
20170419:01:10:35:116684 gpinitsystem:gssapj-dev2:gpadmin-[INFO]:-Checking Master host
20170419:01:10:35:116684 gpinitsystem:gssapj-dev2:gpadmin-[INFO]:-Checking new segment hosts, please wait...
..
20170419:01:10:38:116684 gpinitsystem:gssapj-dev2:gpadmin-[INFO]:-Checking new segment hosts, Completed
20170419:01:10:38:116684 gpinitsystem:gssapj-dev2:gpadmin-[INFO]:-Building the Master instance database, please wait...
20170419:01:10:47:116684 gpinitsystem:gssapj-dev2:gpadmin-[INFO]:-Starting the Master in admin mode
20170419:01:10:54:gpinitsystem:gssapj-dev2:gpadmin-[FATAL]:-Failed to complete obtain psql count Master gp_segment_configuration Script Exiting!
20170419:01:10:54:116684 gpinitsystem:gssapj-dev2:gpadmin-[WARN]:-Script has left Greenplum Database in an incomplete state
20170419:01:10:54:116684 gpinitsystem:gssapj-dev2:gpadmin-[WARN]:-Run command /bin/bash /home/gpadmin/gpAdminLogs/backout_gpinitsystem_gpadmin_20170419_011033 to remove these changes
20170419:01:10:54:116684 gpinitsystem:gssapj-dev2:gpadmin-[INFO]:-Start Function BACKOUT_COMMAND
20170419:01:10:54:116684 gpinitsystem:gssapj-dev2:gpadmin-[INFO]:-End Function BACKOUT_COMMAND
And the following error messages are also seen in the Greenplum log file:

2017-04-19 01:10:54.238111 EDT,"gpadmin","template1",p122240,th116934464,"[local]",,2017-04-19 01:10:54 EDT,0,,,seg-1,,,,,"FATAL","XX000","semctl(58851360, 12, SETVAL, 0) failed: Invalid argument (pg_sema.c:151)",,,,,,,0,,"pg_sema.c",151,"Stack trace:
1 0x943c3b postgres errstart (elog.c:520)
2 0x4c2699 postgres <symbol not found> (pg_sema.c:145)
3 0x7b55c5 postgres <symbol not found> (pg_sema.c:371)
4 0x816eb3 postgres InitProcess (proc.c:426)
5 0x82964f postgres PostgresMain (postgres.c:4607)
6 0x7cb0c5 postgres <symbol not found> (postmaster.c:6366)
7 0x7cd4e2 postgres PostmasterMain (postmaster.c:1523)
8 0x7053e7 postgres main (main.c:206)
9 0x7f2800411b15 libc.so.6 __libc_start_main + 0xf5
10 0x4c4c95 postgres <symbol not found> + 0x4c4c95
"
......
2017-04-19 01:10:54.245745 EDT,,,p121300,th116934464,,,,0,,,seg-1,,,,,"LOG","00000","semctl(58818590, 0, IPC_RMID, ...) failed: Invalid argument",,,,,,,0,,"pg_sema.c",165,
2017-04-19 01:10:54.245781 EDT,,,p121300,th116934464,,,,0,,,seg-1,,,,,"LOG","00000","semctl(58851360, 0, IPC_RMID, ...) failed: Invalid argument",,,,,,,0,,"pg_sema.c",165,
2017-04-19 01:10:54.245806 EDT,,,p121300,th116934464,,,,0,,,seg-1,,,,,"LOG","00000","semctl(58884129, 0, IPC_RMID, ...) failed: Invalid argument",,,,,,,0,,"pg_sema.c",165,

Environment

Cause

The IPC objects (like shared memory/semaphore) were cleaned up by the operating system automatically in the background.

Since systemd service was introduced into Redhat 7.x (also CentOS 7.x), it will clean up the IPC objects when a user fully logs out if "RemoveIPC=yes" (default is yes) is set in /etc/systemd/logind.conf.

 

Resolution

Follow the steps to resolve this issue:

1. Create the gpadmin user as a system account
2. Disable RemoveIPC

Refer to Configuring Your Systems and Installing Greenplum for more details.