Full recovery(gprecoverseg) failed with error "Failed to inform primary segment of updated mirroring state" after running for 60 minutes.

Products

• VMware Tanzu Greenplum

Issue/Introduction

If the full recovery is still running in backend, then there is no need to change the segment status and rerun full recovery.

Symptoms:
When running full recovery for huge instances, for example in an instance with over 800M files inside, the full recovery may not be able to complete in 60 minutes. It may fail with the error below:

20190308:18:10:47:144528 gprecoverseg:mdw:gpadmin-[INFO]:-Commencing parallel primary conversion of 1 segments, please wait...
20190308:19:10:49:144528 gprecoverseg:mdw:gpadmin-[INFO]:-Process results...
20190308:19:10:49:144528 gprecoverseg:mdw:gpadmin-[WARNING]:-Failed to inform primary segment of updated mirroring state.  Segment: sdwxx:/dataxx/mirror/gpsegxx:content=xxx:dbid=xxx:mode=r:status=u: REASON: Conversion failed.  stdout:""  stderr:"failure: timeout  Retrying no 1  failure: OtherTransitionInProgress failure: OtherTransitionInProgress "
20190308:19:10:49:144528 gprecoverseg:mdw:gpadmin-[INFO]:-Done updating primaries

Environment

Product Version: 5.17

Resolution

Checklist:
If full recovery fails, we are still able to run gpstate -e in order to check if it's still running.

Check whether the Objects to resync column is still moving.

If it is moving, then we know the full recovery is still running through recovery process. Once this column becomes 0, it will move to the next stage and report the real progress and the total data size will be synced.

20190308:16:58:47:132460 gpstate:mdw:gpadmin-[INFO]:-Segment Pairs in Resynchronization
20190308:16:58:47:132460 gpstate:mdw:gpadmin-[INFO]:-   Current Primary   Port    Resync mode   Est. resync progress   Total resync objects   Objects to resync   Data synced   Est. total to sync   Est. resync end time   Change tracking size   Mirror   Port
20190308:16:58:47:132460 gpstate:mdw:gpadmin-[INFO]:-   sdw12             50001   Full          Not Available          8369871                2590813             0 bytes       Not Available        Not Available          100 MB                 sdw11    40001