gprestore with --resize-cluster will fail with "No such file or directory" if --jobs > 1

Products

• VMware Tanzu Greenplum
• Greenplum
• Pivotal Data Suite Non Production Edition
• VMware Tanzu Data Suite
• VMware Tanzu Data Suite

Issue/Introduction

When restoring data from different cluster, we can use --resize-cluster option.
However, it may fail if it is run with --jobs set to greater than 1. Example log with error:

$ gprestore --jobs 2 --resize-cluster --backup-dir /data/nas_test --verbose
20240930:14:11:40 gprestore:gpadmin:matt-gpdb-aio:009841-[DEBUG]:-Restore Command: [gprestore --jobs 2 --resize-cluster --backup-dir /data/nas_test --verbose]
20240930:14:11:40 gprestore:gpadmin:matt-gpdb-aio:009841-[INFO]:-Restore Key = 20240930135837
20240930:14:11:40 gprestore:gpadmin:matt-gpdb-aio:009841-[INFO]:-Resize restore specified, will restore a backup set from a 4-segment cluster to a 2-segment cluster
20240930:14:11:40 gprestore:gpadmin:matt-gpdb-aio:009841-[INFO]:-gpbackup version = 1.30.5
20240930:14:11:40 gprestore:gpadmin:matt-gpdb-aio:009841-[INFO]:-gprestore version = 1.30.5
20240930:14:11:40 gprestore:gpadmin:matt-gpdb-aio:009841-[INFO]:-Greenplum Database Version = 6.26.3 build commit:6eeb38781a1816e42a610af0137c45657bcf6960
...
20240930:14:11:41 gprestore:gpadmin:matt-gpdb-aio:009841-[DEBUG]:-Creating segment data pipes
20240930:14:11:41 gprestore:gpadmin:matt-gpdb-aio:009841-[DEBUG]:-Creating segment data pipes
20240930:14:11:41 gprestore:gpadmin:matt-gpdb-aio:009841-[DEBUG]:-Starting gpbackup_helper agent
20240930:14:11:41 gprestore:gpadmin:matt-gpdb-aio:009841-[DEBUG]:-Executing "COPY public.aaa(a) FROM PROGRAM 'cat <SEG_DATA_DIR>/gpbackup_<SEGID>_20240930135837_pipe_9841_741959_0' WITH CSV DELIMITER ',' ON SEGMENT;" on master
20240930:14:11:41 gprestore:gpadmin:matt-gpdb-aio:009841-[DEBUG]:-Executing "COPY public.temp01(id,now) FROM PROGRAM 'cat <SEG_DATA_DIR>/gpbackup_<SEGID>_20240930135837_pipe_9841_750146_0' WITH CSV DELIMITER ',' ON SEGMENT;" on master
20240930:14:11:41 gprestore:gpadmin:matt-gpdb-aio:009841-[DEBUG]:-Executing "COPY public.aaa(a) FROM PROGRAM 'cat <SEG_DATA_DIR>/gpbackup_<SEGID>_20240930135837_pipe_9841_741959_1' WITH CSV DELIMITER ',' ON SEGMENT;" on master
20240930:14:11:41 gprestore:gpadmin:matt-gpdb-aio:009841-[DEBUG]:-Restored data to table public.aaa from file (table 1 of 31)
20240930:14:11:41 gprestore:gpadmin:matt-gpdb-aio:009841-[DEBUG]:-Executing "COPY public.big_table_32429(id,c1,c2,c3,c4,c5,c6,c7,c8,c9,c10,c11,c12,c13,c14,c15) FROM PROGRAM 'cat <SEG_DATA_DIR>/gpbackup_<SEGID>_20240930135837_pipe_9841_750227_0' WITH CSV DELIMITER ',' ON SEGMENT;" on master
20240930:14:11:41 gprestore:gpadmin:matt-gpdb-aio:009841-[ERROR]:-Error loading data into table public.temp01: ERROR: command error message: cat: /data/segment/segment_6.26.3/primary1/gpdb_6.26.3_0/gpbackup_0_20240930135837_pipe_9841_750146_0: No such file or directory  (seg0 x.x.x.x:xxx pid=9870) (SQLSTATE 2F000)

Resolution

Fix

This is a known issue of gprestore and has been fixed in version 1.30.6 and above.

See Release Notes for gprestore:

gprestore v1.30.6
Resolved Issues
...
Resolves an issue where gprestore with --resize-cluster and either --copy-queue-size or --jobs would fail with the error No such file or directory.

Workaround

For versions older than 1.30.6: set the --jobs to 1 to workaround the issue.