VMware Tanzu Greenplum 6's gprecoverseg explained
Article ID: 296300
Updated On:
Products
VMware Tanzu Greenplum
Issue/Introduction
gprecoverseg in VMware Tanzu Greenplum 6 has changed in a few different ways. This article explains how gprecoverseg works in VMware Tanzu Greenplum 6, how you should use it and what you should expect to see.
Environment
Product Version: 6.0
Resolution
In VMware Tanzu Greenplum 6 gprecoverseg works differently for Full and Incremental recoverys. So lets look at each one separately:
A full recovery uses pg_basebackup. Firstly it will create backup primary segments, then recreate the mirror segment, so that the mirror is now empty (in VMware Tanzu Greenplum 4 and VMware Tanzu Greenplum 5 it used to delete the segment files). Then it will send the backup to the mirror segment and write the data from the backup to the new segment. Finally, it will start the segment and mark it up.
The output will give a progress report of the files being copied to the mirror segment and a report done when it is complete. By default, the progress is replaced by the new progression report and then by Done when it is complete.
Using the -s option will permanently print the progression report to screen and the log file. Here is an example of a very small recovery:
'Not In Sync' means that the segment is still in resync state. The recovery is complete when the segment Data Status is in 'Synchronized'.
To estimate how much data is left to resycn, you will need to use gpstate -s and go to the segment that was recovered. The 'Mirror status' in the primary segment will be 'Not In Sync' and the 'WAL Replay Location' under 'Replication Info' of the mirror will tell you how many bytes are left to sync.
Full Recovery
A full recovery uses pg_basebackup. Firstly it will create backup primary segments, then recreate the mirror segment, so that the mirror is now empty (in VMware Tanzu Greenplum 4 and VMware Tanzu Greenplum 5 it used to delete the segment files). Then it will send the backup to the mirror segment and write the data from the backup to the new segment. Finally, it will start the segment and mark it up. The output will give a progress report of the files being copied to the mirror segment and a report done when it is complete. By default, the progress is replaced by the new progression report and then by Done when it is complete.
Using the -s option will permanently print the progression report to screen and the log file. Here is an example of a very small recovery:
20210407:13:20:33:014421 gprecoverseg:gpdb-tablespace-m:gpadmin-[INFO]:-Starting to modify pg_hba.conf on primary segments to allow replication connections 20210407:13:20:39:014421 gprecoverseg:gpdb-tablespace-m:gpadmin-[INFO]:-Successfully modified pg_hba.conf on primary segments to allow replication connections 20210407:13:20:39:014421 gprecoverseg:gpdb-tablespace-m:gpadmin-[INFO]:-1 segment(s) to recover 20210407:13:20:39:014421 gprecoverseg:gpdb-tablespace-m:gpadmin-[INFO]:-Ensuring 1 failed segment(s) are stopped 20210407:13:20:40:014421 gprecoverseg:gpdb-tablespace-m:gpadmin-[INFO]:-Ensuring that shared memory is cleaned up for stopped segments 20210407:13:20:40:014421 gprecoverseg:gpdb-tablespace-m:gpadmin-[INFO]:-Validating remote directories 20210407:13:20:41:014421 gprecoverseg:gpdb-tablespace-m:gpadmin-[INFO]:-Configuring new segments gpdb-tablespace-1 (dbid 9): 127267/216587 kB (58%), 0/2 tablespaces (...GPDB_6_301908232/16384/16384_fsm) gpdb-tablespace-1 (dbid 9): pg_basebackup: base backup completed 20210407:13:20:45:014421 gprecoverseg:gpdb-tablespace-m:gpadmin-[INFO]:-Updating configuration with new mirrors 20210407:13:20:45:014421 gprecoverseg:gpdb-tablespace-m:gpadmin-[INFO]:-Updating mirrors 20210407:13:20:45:014421 gprecoverseg:gpdb-tablespace-m:gpadmin-[INFO]:-Starting mirrors 20210407:13:20:45:014421 gprecoverseg:gpdb-tablespace-m:gpadmin-[INFO]:-era is f60ff30cbbc77b9d_210405112734 20210407:13:20:45:014421 gprecoverseg:gpdb-tablespace-m:gpadmin-[INFO]:-Commencing parallel segment instance startup, please wait...
The gprecoverseg for a full recovery completes when the data is fully copied from primary to mirror. Therefore, there is no need to monitor the resync of data for a full recovery using gpstate.
However, this will mean that the gprecoverseg process will not take longer to complete compared to VMware Tanzu Greenplum 4 and VMware Tanzu Greenplum 5, but the overall recovery (including the data resync) is quicker.
Incremental Recovery
An Incremental recovery uses pg_rewind to copy the contents that is in the changetracking log.
Firstly, the pg_rewind does a scan of all of the files on the primary.
Note: We have seen that this process can take a long time to run if you see the dots after the statement.
[INFO]:-Running pg_rewind on required mirrors
Then you can monitor the pg_rewind scan with the following:
+ Check of the pg_rewind process
gpssh -f hostfile -e "ps -ef | grep rewind"
Once all of the rewind processes are done, the recovery will move to the next step.
To verify it is still working :
+ Run "strace -p <rewind process number>"
You will see and each lstats is taking sometime to move to the next
lstat("<PATH / filenumber>", {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
In VMware Tanzu Greenplum 6.14 and after, we should see the progress of the rewind in percentage form
For example:
20210111:08:52:52:021213 gprecoverseg:mdw-[INFO]:-Running pg_rewind on required mirrors mdw.vmware.com (dbid 2): Done! mdw.vmware.com (dbid 3): 50478/213442 kB (23%) copied mdw.vmware.com (dbid 2): Done! mdw.vmware.com (dbid 3): Done! 20210111:08:52:56:021213 gprecoverseg:mdw-[INFO]:-Starting mirrors
Then it will start the mirror and begin to resync the data from the primary to the mirror.
The gprecoverseg process completes when the mirror segment is up and when the resync process begins. Therefore, once the gprecoverseg is done, you will have to use gpstate to monitor the resync of the data.
Use gpstate -m to get summary segment 'Data Status':
20210408:11:57:09:000564 gpstate:mdw-lab1:gpadmin-[INFO]:-------------------------------------------------------------- 20210408:11:57:09:000564 gpstate:mdw-lab1:gpadmin-[INFO]:- Mirror Datadir Port Status Data Status 20210408:11:57:09:000564 gpstate:mdw-lab1:gpadmin-[INFO]:- sdw2-lab1 /data/mirror/gp_6.15.0_202104031034_someguy_seg0 35070 Passive Synchronized 20210408:11:57:09:000564 gpstate:mdw-lab1:gpadmin-[INFO]:- sdw2-lab1 /data/mirror/gp_6.15.0_202104031034_someguy_seg1 35071 Passive Synchronized 20210408:11:57:09:000564 gpstate:mdw-lab1:gpadmin-[INFO]:- sdw1-lab1 /data/mirror/gp_6.15.0_202104031034_someguy_seg2 35070 Passive Not In Sync 20210408:11:57:09:000564 gpstate:mdw-lab1:gpadmin-[INFO]:- sdw1-lab1 /data/mirror/gp_6.15.0_202104031034_someguy_seg3 35071 Passive Not In Sync
'Not In Sync' means that the segment is still in resync state. The recovery is complete when the segment Data Status is in 'Synchronized'.
To estimate how much data is left to resycn, you will need to use gpstate -s and go to the segment that was recovered. The 'Mirror status' in the primary segment will be 'Not In Sync' and the 'WAL Replay Location' under 'Replication Info' of the mirror will tell you how many bytes are left to sync.
20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:----------------------------------------------------- 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- Segment Info 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- Hostname = sdw2-lab1 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- Address = sdw2-lab1 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- Datadir = /data/primary/gp_6.15.0_202104031034_someguy_seg2 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- Port = 30070 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- Mirroring Info 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- Current role = Primary 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- Preferred role = Primary 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- Mirror status = Not In Sync 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- Status 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- PID = 16310 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- Configuration reports status as = Up 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- Database status = Up 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:----------------------------------------------------- 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- Segment Info 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- Hostname = sdw1-lab1 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- Address = sdw1-lab1 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- Datadir = /data/mirror/gp_6.15.0_202104031034_someguy_seg2 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- Port = 35070 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- Mirroring Info 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- Current role = Mirror 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- Preferred role = Mirror 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- Mirror status = Streaming 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- Replication Info 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- WAL Sent Location = 3/7C018C08 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- WAL Flush Location = 3/7C018C08 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- WAL Replay Location = 3/716B4E60 (177618344 bytes left) 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- Status 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- PID = 28752 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- Configuration reports status as = Up 20210412:16:27:36:024007 gpstate:mdw-lab1:gpadmin-[INFO]:- Segment status = Up
Feedback
Yes
No
Powered by
