Local Greenplum version not matching Greenplum utility logging/output
search cancel

Local Greenplum version not matching Greenplum utility logging/output

book

Article ID: 397577

calendar_today

Updated On:

Products

VMware Tanzu Data Suite VMware Tanzu Greenplum VMware Tanzu Greenplum / Gemfire

Issue/Introduction

Greenplum 7.x installed locally, however when running gpstart/gpstop output shows 6.x, in addition to failing:

20250506:08:52:14:378170 gpstop:gphost:gpadmin-[INFO]:-Starting gpstop with args: 
20250506:08:52:14:378170 gpstop:gphost:gpadmin-[INFO]:-Gathering information and validating the environment...
20250506:08:52:14:378170 gpstop:gphost:gpadmin-[INFO]:-Obtaining Greenplum Coordinator catalog information
20250506:08:52:14:378170 gpstop:gphost:gpadmin-[INFO]:-Obtaining Segment details from coordinator...
20250506:08:52:14:378170 gpstop:gphost:gpadmin-[CRITICAL]:-gpstop failed. (Reason='FATAL:  the database system is in recovery mode
DETAIL:  last replayed record at A1A/1465A590
- VERSION: PostgreSQL 9.4.26 (Greenplum Database 6.28.0 build commit:dc1f64048a6b086d2a139a5b610ea2b04ec7b6c4) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Aug  8 2024 07:32:28
') exiting...


Additionally psql fails and reports a 'database system is in recovery mode' error:

[gpadmin@gphost ~]$ psql
psql: error: FATAL:  the database system is in recovery mode
DETAIL:  last replayed record at A1A/14A44288
- VERSION: PostgreSQL 9.4.26 (Greenplum Database 6.28.0 build commit:dc1f64048a6b086d2a139a5b610ea2b04ec7b6c4) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Aug  8 2024 07:32:28

Environment

All Linux-based Greenplum environments

Cause

local gp utilities may be attempting to connect to an incorrect instance of Greenplum, using stale metadata, and/or utilizing incorrect environment variables.

Resolution

Greenplum Database versions are not hard coded in to GP utilities such as gpstart/gpstop. If the output of such utilities does not match the target database, This could mean that metadata and/or environment variables should be closely scrutinized to verify everything is correctly set for the target, local or otherwise.

1. Use 'ps aux' to verify that there is only 1 instance of Greenplum/Postgres running on the target host

[gpadmin@gphost ~]$ ps aux | grep -i postgres
gpadmin   509062  0.1  3.5 333828 64296 ?        Ss   12:23   0:00 /usr/local/greenplum-db-7.4.0/bin/postgres -D /data/coordinator/gpseg-1 -c gp_role=dispatch
gpadmin   509063  0.0  0.5 139152 10420 ?        Ss   12:23   0:00 postgres:  5432, master logger process   
gpadmin   509065  0.0  0.5 333828 10720 ?        Ss   12:23   0:00 postgres:  5432, checkpointer   
gpadmin   509066  0.0  0.6 334128 11736 ?        Ss   12:23   0:00 postgres:  5432, background writer   
gpadmin   509067  0.0  0.8 333992 15616 ?        Ss   12:23   0:00 postgres:  5432, walwriter   
gpadmin   509068  0.0  0.9 336320 17428 ?        Ss   12:23   0:00 postgres:  5432, autovacuum launcher   
gpadmin   509069  0.0  0.5 141572 10580 ?        Ss   12:23   0:00 postgres:  5432, stats collector   
gpadmin   509070  0.1  1.5 403736 27948 ?        Ssl  12:23   0:00 postgres:  5432, dtx recovery process   
gpadmin   509071  0.0  1.4 403544 26124 ?        Ssl  12:23   0:00 postgres:  5432, ftsprobe process   
gpadmin   509075  0.0  0.8 336240 15472 ?        Ss   12:23   0:00 postgres:  5432, logical replication launcher   
gpadmin   509076  0.0  0.6 333984 12584 ?        Ss   12:23   0:00 postgres:  5432, ic proxy process   
gpadmin   509078  0.1  0.5 333988 10612 ?        Ss   12:23   0:00 postgres:  5432, metrics collector   
gpadmin   509088  0.0  0.0  12216  1100 pts/0    S+   12:24   0:00 grep --color=auto -i postgres

 

2. With the postgres process ID (509062 in this case) from 'ps aux' output use a network utility such as 'ss' or 'netstat' to verify that the assigned Postgres port (default: 5432) is being used by the Postgres process:

gpadmin@gphost ~]$ netstat -tunlp
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:5432            0.0.0.0:*               LISTEN      509062/postgres     
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      -                   
tcp6       0      0 :::22                   :::*                    LISTEN      -                   
tcp6       0      0 :::5432                 :::*                    LISTEN      509062/postgres     
tcp6       0      0 :::111                  :::*                    LISTEN      -                   
udp        0      0 10.159.240.240:41667    0.0.0.0:*                           509071/postgres:  5 
udp        0      0 10.159.240.240:43736    0.0.0.0:*                           509070/postgres:  5 
udp        0      0 10.159.240.240:51991    0.0.0.0:*                           509070/postgres:  5 
udp        0      0 0.0.0.0:111             0.0.0.0:*                           -                   
udp        0      0 10.159.240.240:38015    0.0.0.0:*                           509071/postgres:  5 
udp6       0      0 :::111                  :::*                                -

 

3. Check the data directory for the existence of stale lock directories & verify that postmaster.pid contains the correct Postgres process ID:

[gpadmin@gphost gpseg-1]$ cat postmaster.pid 
509062
/data/coordinator/gpseg-1
1747250639
5432
/tmp
*
  5432001     32782
dtmready

 

4. Finally, check for any erroneous environment variables:

[gpadmin@gphost ~]$ env
LD_LIBRARY_PATH=/usr/local/greenplum-db-7.4.0/lib
LS_COLORS=rs=0:di=38;5;33:ln=38;5;51:mh=00:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=01;05;37;41:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;40:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.zst=38;5;9:*.tzst=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.wim=38;5;9:*.swm=38;5;9:*.dwm=38;5;9:*.esd=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.mjpg=38;5;13:*.mjpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.m4a=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.oga=38;5;45:*.opus=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
LANG=en_US.UTF-8
HISTCONTROL=ignoredups
HOSTNAME=rocky1
JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.442.b06-2.el8.x86_64/jre
S_COLORS=auto
which_declare=declare -f
USER=gpadmin
PWD=/home/gpadmin
HOME=/home/gpadmin
GPHOME=/usr/local/greenplum-db-7.4.0
COORDINATOR_DATA_DIRECTORY=/data/coordinator/gpseg-1
MAIL=/var/spool/mail/gpadmin
SHELL=/bin/bash
TERM=xterm-256color
SHLVL=1
PYTHONPATH=/usr/local/greenplum-db-7.4.0/lib/python
LOGNAME=gpadmin
GPCC_HOME=/usr/local/greenplum-cc-7.2.0
PATH=/usr/local/greenplum-cc-7.2.0/bin:/usr/local/greenplum-db-7.4.0/bin:/home/gpadmin/.local/bin:/home/gpadmin/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
DEBUGINFOD_URLS=https://debuginfod.centos.org/ 
HISTSIZE=1000
LESSOPEN=||/usr/bin/lesspipe.sh %s
PGPORT=5432
PGHOST=prod.db.com
BASH_FUNC_which%%=() {  ( alias;
 eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot $@
}
_=/usr/bin/env
OLDPWD=/home/gpadmin

 

From the environment output, I see that two Postgres related variables (PGHOST, PGPORT) are set, which typically are not used in Greenplum. While it is ok to have these set, its important to understand that these will override any values that some GP utilities may use.

PGPORT is correctly set to 5432, however PGHOST is not set to the local 'gphost' hostname. To fix this, do one of either:

unset PGHOST

#or

export PGHOST=gphost

Additional Information

Using gpstop with -v would verify which host gpstop is attempting to call:

20250506:09:36:19:380937 gpstop:gphost:gpadmin-[INFO]:-Starting gpstop with args: -v
20250506:09:36:19:380937 gpstop:gphost:gpadmin-[DEBUG]:-Setting level of parallelism to: 64
20250506:09:36:19:380937 gpstop:gphost:gpadmin-[INFO]:-Gathering information and validating the environment...
20250506:09:36:19:380937 gpstop:gphost:gpadmin-[DEBUG]:---Checking that current user can use GP binaries
20250506:09:36:19:380937 gpstop:gphost:gpadmin-[DEBUG]:-Obtaining coordinator's port from coordinator data directory
20250506:09:36:19:380937 gpstop:gphost:gpadmin-[DEBUG]:-Read from postgresql.conf port=5432
20250506:09:36:19:380937 gpstop:gphost:gpadmin-[DEBUG]:-Connecting to db template1 on host pe1uautogpam02.res.prod.global
20250506:09:36:19:380937 gpstop:gphost:gpadmin-[INFO]:-Obtaining Greenplum Coordinator catalog information
20250506:09:36:19:380937 gpstop:gphost:gpadmin-[INFO]:-Obtaining Segment details from coordinator...
20250506:09:36:19:380937 gpstop:gphost:gpadmin-[DEBUG]:-Running Command: $GPHOME/sbin/gpconfig_helper.py --file /data/coordinator/gpseg-1/postgresql.conf --get-parameter gp_segment_configuration_file
20250506:09:36:19:380937 gpstop:gphost:gpadmin-[DEBUG]:-Connecting to db template1 on host prod.db.com
20250506:09:36:19:380937 gpstop:gphost:gpadmin-[ERROR]:-gpstop failed.  exiting...
Traceback (most recent call last):
  File "/usr/local/greenplum-db-7.4.0/lib/python/gppylib/mainUtils.py", line 361, in simple_main_locked
    exitCode = commandObject.run()
  File "/usr/local/greenplum-db-7.4.0/bin/gpstop", line 162, in run
    self._prepare()
  File "/usr/local/greenplum-db-7.4.0/bin/gpstop", line 260, in _prepare
    self._build_gparray()
  File "/usr/local/greenplum-db-7.4.0/bin/gpstop", line 318, in _build_gparray
    self.gparray = GpArray.initFromGpSegmentFileOrCatalog(self.dburl, gp_segment_config_file, utility=True)
  File "/usr/local/greenplum-db-7.4.0/lib/python/gppylib/gparray.py", line 1092, in initFromGpSegmentFileOrCatalog
    return GpArray.initFromCatalog(dbURL, utility)
  File "/usr/local/greenplum-db-7.4.0/lib/python/gppylib/gparray.py", line 990, in initFromCatalog
    with closing(dbconn.connect(dbURL, utility)) as conn:
  File "/usr/local/greenplum-db-7.4.0/lib/python/gppylib/db/dbconn.py", line 238, in connect
    conn = psycopg2.connect(**conninfo)
  File "/usr/lib64/python3.9/site-packages/psycopg2/__init__.py", line 127, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL:  the database system is in recovery mode
DETAIL:  last replayed record at A1A/2818EA08
- VERSION: PostgreSQL 9.4.26 (Greenplum Database 6.28.0 build commit:dc1f64048a6b086d2a139a5b610ea2b04ec7b6c4) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Aug  8 2024 07:32:28
Powered by Wolken Software