DDBoost write failed on gpbackup_57_20220330090733.gz [DDBoost Error 5057: File handle is stale]

Products

• VMware Tanzu Greenplum

Issue/Introduction

Backup of a database fails when using gpbackup to backup to a DataDomain using DDBoost plugin and the option "--single-data-file".

The errors in the gpbackup log on the master are similar to:

20220330:16:25:55 gpbackup:gpadmin:mdw1.mgmt:184874-[ERROR]:-Encountered errors with 17 helper agent(s).  See /home/gpadmin/gpAdminLogs/gpbackup_20220330.log for a complete list of segments with errors, and see /home/gpadmin/gpAdminLogs/gpbackup_helper_20220330.log on the corresponding hosts for detailed error messages.
20220330:16:25:55 gpbackup:gpadmin:mdw1.mgmt:184874-[CRITICAL]:-ERROR: Error from segment 1: ERROR:  command error message: (SQLSTATE 22P04)

The errors reported in the gpbackup_helper log files on the segment servers are similar to:

16:25:55 ERROR ddboost_plugin.c:1708: DDBoost write failed on /test/GPDBBACKUP/20220330/20220330090733/gpbackup_1_20220330090733.gz [DDBoost Error 5057: File handle is stale]

This can occur when some segments complete the backup of a table quicker than other segments.
The segments that have completed will not write any further data to the DataDomain until all segments complete the backup of the current table and gpbackup moves on to the next table to be backed up.
If a segment completes the backup of the table and does not write any data to the DataDomain for a time longer than "OST_ABANDON_TIMEOUT", then the DataDomain will close the file and cause Greenplum backup to fail.

Environment

Product Version: 5.28

Resolution

Increase the OST_ABANDON_TIMEOUT on the DataDomain.
See Dell Technologies KB for more information.