[gprecoverseg] Incremental recovery failed for dbid XX. You must use gprecoverseg -F to recover the segment

Products

• VMware Tanzu Greenplum

Issue/Introduction

When running gprecoverseg to recover down segment(s), we might observe the error below:

20200817:03:12:11:006956 gprecoverseg:mdw:gpadmin-[WARNING]:-Incremental recovery failed for dbid 4. You must use gprecoverseg -F to recover the segment.

If we run gprecoverseg with verbose mode (gprecoverseg -v), then it will give us more details - the xlog file is missing:

20200817:03:12:11:006956 gprecoverseg:mdw:gpadmin-[DEBUG]:-pg_rewind results: cmd had rc=1 completed=True halted=False stdout='connected to server servers diverged at WAL position 1/91477F98 on timeline 5 could not open file "/data2/segment/segment_6.9.1/primary1/gpseg2/pg_xlog/00000005000000010000001E": No such file or directory could not find previous WAL record at 1/7BFFFE80 Failure, exiting

This issue only existed in GPDB v6, and has been fixed after 6.11.

Environment

Product Version: 6.7

Resolution

Based on the investigation from the R&D team, this is caused by a known issue in GPDB v6 which is before 6.11. 

Here is one of the scenarios that may trigger this issue:

1. Multiple data loading jobs are running in parallel
2. Reboot on of the segment server (server crash)
3. Running gprecoverseg 

The R&D team has fixed the issue in GPDB v6.11 and afterward.
Please refer to the release note for more details: https://gpdb.docs.pivotal.io/6-11/relnotes/gpdb-611-release-notes.html

30870 - Segment Mirroring
In some cases, performing an incremental recovery of a Greenplum Database segment instance failed with the message requested WAL segment has already been removed because the recovery checkpoint was not created properly. This issue is resolved.

Workaround:

The workaround for this issue is to do the full recovery (gprecoverseg -F) to bring up the down segments.