"unexpected control file size 0, expected 8192" error during Incremental Recovery and recommends full recovery in Tanzu Greenplum

Products

• VMware Tanzu Greenplum

Issue/Introduction

When running an increment recovery on Tanzu Greenplum 6.x, it fails and recommends to run a full recovery. 

[WARNING]:-Incremental recovery failed for dbid 37. You must use gprecoverseg -F to recover the segment.

The recovery logs may also report failure to the access file:

20211206:16:37:54:010338 gprecoverseg:mdw:gpadmin-[WARNING]:-servers diverged at WAL position 9D3/937FEBE0 on timeline 3
rewinding from last common checkpoint at 9D3/79557DF0 on timeline 3

could not open target file "/data/primary/gpseg18/gpbackup_history.yaml": Permission denied
Failure, exiting

When you run the recovery in verbose mode, you should see the following error:

stdout='connected to server

unexpected control file size 0, expected 8192
Failure, exiting
'
  stderr=''

Environment

Product Version: 6.12

Resolution

This is caused by a bug in the Tanzu Greenplum 6.x. Incremental recovery in Greenplum had a bug that manifests when a failed segment is acting as the primary before failure and recovery.

Recovering a failed primary incrementally involves detecting a checkpoint in its WAL that is also found in its peer segment's WAL. This is known as point of divergence. A bug in the logic to remove old WAL files caused the WAL file containing the point of divergence to be removed when it is still needed for incremental recovery. 

To work around this issue, you have to do a full recovery of the segments. 

This bug is fixed in Tanzu Greenplum 6.16.1 or above, and the WAL file is properly retained. To avoid this issue going forward, upgrade to Tanzu Greenplum 6.16.1 or above.