gprestore hangs while restoring a large backup and using --resize-cluster option

Products

• VMware Tanzu Greenplum
• Greenplum
• Pivotal Data Suite Non Production Edition
• VMware Tanzu Data Suite
• VMware Tanzu Data Suite

Issue/Introduction

When restoring a large backup the gprestore can stop restoring the data and hang forever.

If the restore is to a smaller cluster than the original (using --resize-cluster), the issue is more likely to occur.

the gpbackup_helper log file on the segment servers may show:

20241116:02:06:29 gpbackup_helper:gpadmin:XEMDUB07DBXP0B0:306802-[DEBUG]:-Segment 5: Waiting to open writer to pipe /data2/primary/gpseg5/gpbackup_5_20241115193032_pipe_3768883_730722 because reader has not yet tried to access it. Have been waiting for: 1m21.91s
20241116:02:07:29 gpbackup_helper:gpadmin:XEMDUB07DBXP0B0:306802-[DEBUG]:-Segment 5: Waiting to open writer to pipe /data2/primary/gpseg5/gpbackup_5_20241115193032_pipe_3768883_730722 because reader has not yet tried to access it. Have been waiting for: 2m21.91s
20241116:02:08:29 gpbackup_helper:gpadmin:XEMDUB07DBXP0B0:306802-[DEBUG]:-Segment 5: Waiting to open writer to pipe /data2/primary/gpseg5/gpbackup_5_20241115193032_pipe_3768883_730722 because reader has not yet tried to access it. Have been waiting for: 3m21.91s
   :

It is OK if the above message is repeated for a short period (approx 10-15 minutes). It may be due to heavy load on the system, but it can get locked up in an infinite loop.

Resolution

Workaround

It may help to reduce the "--jobs" option to a lower number to reduce load.

Fix

The issue has been fixed in gpbackup 1.30.7 and above