How to reboot standby master

Products

• VMware Tanzu Greenplum

Issue/Introduction

For a maintenance purposes you might need to reboot a standby master. This process does not require gpdb to be shutdown or reboot.

Environment

Product Version: 5.23

Resolution

1. ssh to standby master and get the information about currently running postgres process (you will need the -D path variable):

[gpadmin@mdw_lab2 ~]$ ssh smdw_lab2
Last login: Fri Jun 12 10:59:12 2020 from mdw_lab2
[gpadmin@smdw_lab2 ~]$ ps -elf | grep silent
1 S gpadmin  10710     1  0  80   0 - 110372 poll_s 11:06 ?       00:00:00 /usr/local/greenplum-db-5.21.1/bin/postgres -D /data/master/gp_5.21.1_202005180902_kris_seg-1 -p 3512 --gp_dbid=10 --gp_num_contents_in_cluster=4 --silent-mode=true -i -M master --gp_contentid=-1 -x 0 -y -E

2. Source the greenplum path to have access to pg_ctl command and manually shut down the standby process:

[gpadmin@smdw_lab2 ~]$ source /usr/local/greenplum-db/greenplum_path.sh  
[gpadmin@smdw_lab2 ~]$
[gpadmin@smdw_lab2 ~]$ pg_ctl stop -D /data/master/gp_5.21.1_202005180902_kris_seg-1
waiting for server to shut down.... done
server stopped

3. Now when you go back to master and check the state you will see it is not running:

[gpadmin@mdw_lab2 ~]$ gpstate -f
20200612:11:11:12:012764 gpstate:mdw_lab2:gpadmin-[INFO]:-Starting gpstate with args: -f
20200612:11:11:13:012764 gpstate:mdw_lab2:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.21.1 build commit:ca0b8106b893028d18f241dcb858d85f12af90b6'
20200612:11:11:23:012764 gpstate:mdw_lab2:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.21.1 build commit:ca0b8106b893028d18f241dcb858d85f12af90b6) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Jul 24 2019 22:48:12'
20200612:11:11:23:012764 gpstate:mdw_lab2:gpadmin-[INFO]:-Obtaining Segment details from master...
20200612:11:11:33:012764 gpstate:mdw_lab2:gpadmin-[INFO]:-Standby master details
20200612:11:11:33:012764 gpstate:mdw_lab2:gpadmin-[INFO]:-----------------------
20200612:11:11:33:012764 gpstate:mdw_lab2:gpadmin-[INFO]:-   Standby address          = smdw_lab2
20200612:11:11:33:012764 gpstate:mdw_lab2:gpadmin-[INFO]:-   Standby data directory   = /data/master/gp_5.21.1_202005180902_kris_seg-1
20200612:11:11:33:012764 gpstate:mdw_lab2:gpadmin-[INFO]:-   Standby port             = 3512
20200612:11:11:33:012764 gpstate:mdw_lab2:gpadmin-[WARNING]:-Standby PID              = 0                                                <<<<<<<<
20200612:11:11:33:012764 gpstate:mdw_lab2:gpadmin-[WARNING]:-Standby status           = Standby process not running                      <<<<<<<<
20200612:11:11:33:012764 gpstate:mdw_lab2:gpadmin-[INFO]:--------------------------------------------------------------
20200612:11:11:33:012764 gpstate:mdw_lab2:gpadmin-[INFO]:--pg_stat_replication
20200612:11:11:33:012764 gpstate:mdw_lab2:gpadmin-[INFO]:--------------------------------------------------------------
20200612:11:11:43:012764 gpstate:mdw_lab2:gpadmin-[INFO]:-No entries found.
20200612:11:11:43:012764 gpstate:mdw_lab2:gpadmin-[INFO]:--------------------------------------------------------------

4. Reboot your standby master host.

5. gpinitstandby has option -n  executed form master (Specify this option to start a Greenplum Database standby master that has been configured but has stopped for some reason):

[gpadmin@mdw_lab2 ~]$ gpinitstandby -n
20200612:11:13:21:013119 gpinitstandby:mdw_lab2:gpadmin-[INFO]:-Starting standby master
20200612:11:13:21:013119 gpinitstandby:mdw_lab2:gpadmin-[INFO]:-Checking if standby master is running on host: smdw_lab2  in directory: /data/master/gp_5.21.1_202005180902_kris_seg-1
.
20200612:11:13:33:013119 gpinitstandby:mdw_lab2:gpadmin-[INFO]:-Successfully started standby master

6. Now when you check the status everything should be fine:

[gpadmin@mdw_lab2 ~]$ gpstate -f
20200612:11:13:48:013288 gpstate:mdw_lab2:gpadmin-[INFO]:-Starting gpstate with args: -f
20200612:11:13:48:013288 gpstate:mdw_lab2:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.21.1 build commit:ca0b8106b893028d18f241dcb858d85f12af90b6'
20200612:11:13:58:013288 gpstate:mdw_lab2:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.21.1 build commit:ca0b8106b893028d18f241dcb858d85f12af90b6) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Jul 24 2019 22:48:12'
20200612:11:13:58:013288 gpstate:mdw_lab2:gpadmin-[INFO]:-Obtaining Segment details from master...
20200612:11:14:09:013288 gpstate:mdw_lab2:gpadmin-[INFO]:-Standby master details
20200612:11:14:09:013288 gpstate:mdw_lab2:gpadmin-[INFO]:-----------------------
20200612:11:14:09:013288 gpstate:mdw_lab2:gpadmin-[INFO]:-   Standby address          = smdw_lab2
20200612:11:14:09:013288 gpstate:mdw_lab2:gpadmin-[INFO]:-   Standby data directory   = /data/master/gp_5.21.1_202005180902_kris_seg-1
20200612:11:14:09:013288 gpstate:mdw_lab2:gpadmin-[INFO]:-   Standby port             = 3512
20200612:11:14:09:013288 gpstate:mdw_lab2:gpadmin-[INFO]:-   Standby PID              = 11616
20200612:11:14:09:013288 gpstate:mdw_lab2:gpadmin-[INFO]:-   Standby status           = Standby host passive
20200612:11:14:09:013288 gpstate:mdw_lab2:gpadmin-[INFO]:--------------------------------------------------------------
20200612:11:14:09:013288 gpstate:mdw_lab2:gpadmin-[INFO]:--pg_stat_replication
20200612:11:14:09:013288 gpstate:mdw_lab2:gpadmin-[INFO]:--------------------------------------------------------------
20200612:11:14:19:013288 gpstate:mdw_lab2:gpadmin-[INFO]:--WAL Sender State: streaming
20200612:11:14:19:013288 gpstate:mdw_lab2:gpadmin-[INFO]:--Sync state: sync
20200612:11:14:19:013288 gpstate:mdw_lab2:gpadmin-[INFO]:--Sent Location: 0/10000EC0
20200612:11:14:19:013288 gpstate:mdw_lab2:gpadmin-[INFO]:--Flush Location: 0/10000EC0
20200612:11:14:19:013288 gpstate:mdw_lab2:gpadmin-[INFO]:--Replay Location: 0/10000EC0
20200612:11:14:19:013288 gpstate:mdw_lab2:gpadmin-[INFO]:--------------------------------------------------------------