gprestore of an incremental backup fails with "DDBoost open file failed" in Pivotal Greenplum

Products

• VMware Tanzu Greenplum

Issue/Introduction

When using gpbackup or gprestore to take a full backup to Data Domain, it completes successfully. However, when taking an incremental backup after the full backup and then restoring it, gprestore fails with the following error:

20190523:01:26:45 gprestore:gpadmin:hostname.local:049043-[INFO]:-Restore Key = 20190523012138
*** glibc detected *** /usr/local/greenplum-db/./bin/gpbackup_ddboost_plugin: double free or corruption (!prev): 0x0000000000c15b20 ***
======= Backtrace: =========
/lib64/libc.so.6[0x317cc75dee]
/lib64/libc.so.6[0x317cc78c80]
/lib64/libc.so.6(fclose+0x14d)[0x317cc6633d]
/usr/local/greenplum-db/./bin/gpbackup_ddboost_plugin[0x401922]
/usr/local/greenplum-db/./bin/gpbackup_ddboost_plugin[0x4015bf]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x317cc1ed1d]
/usr/local/greenplum-db/./bin/gpbackup_ddboost_plugin[0x401259]
======= Memory map: ========
00400000-00405000 r-xp 00000000 fd:00 4587814              /usr/local/greenplum-db-
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0         [vsyscall]
20190523:01:26:52 gprestore:gpadmin:hostname.local:049043-[CRITICAL]:-signal: aborted (core dumped): DDBoost open file failed on /data/col1/RSGPUATDB1_LOCAL-TEST/20190523/20190523012138/gpbackup_20190523012138_metadata.sql. Err 5004
20190523:01:26:52 gprestore:gpadmin:hostname.local:049043-[INFO]:-/home/gpadmin/gp_email_contacts.yaml list found, /data/master/gpseg-1/backups/20190523/20190523012138/gprestore_20190523012138_20190523012645_report will be sent

Logs for the incremental gpbackup show the following debug information, however it completes successfully:

20190523:16:45:17 gpbackup:gpadmin:hostname.local:041870-[DEBUG]:-Writing filtered oid list to segments runtime error: index out of range 20190523:16:45:18 gpbackup:gpadmin:hostname.local:041870-[INFO]:-/home/gpadmin/gp_email_contacts.yaml list found, /data/master/gpseg-1/backups/20190523/20190523164248/gpbackup_20190523164248_report will be sent 20190523:16:45:18 gpbackup:gpadmin:hostname.local:041870-[DEBUG]:-Sending email report to the following addresses: 20190523:16:45:21 gpbackup:gpadmin:hostname.local:041870-[DEBUG]:-Running plugin cleanup for backup on segment hosts 20190523:16:45:22 gpbackup:gpadmin:hostname.local:041870-[DEBUG]:-Running plugin cleanup for backup on segments 20190523:16:45:24 gpbackup:gpadmin:hostname.local:041870-[DEBUG]:-Beginning cleanup 20190523:16:45:24 gpbackup:gpadmin:hostname.local:041870-[DEBUG]:-Removing oid list and helper script files from segment data directories 20190523:16:45:24 gpbackup:gpadmin:hostname.local:041870-[DEBUG]:-Cleanup complete 20190523:16:45:24 gpbackup:gpadmin:hostname.local:041870-[INFO]:-Backup completed successfully

The file gpbackup_YYYYMMDDHHMMSS_toc.yaml file is not created for the incremental backup operation.

The report file gpbackup_YYYYMMDDHHMMSS_report shows the failed status of the backup:

Backup Status: Failure
Backup Error: or: index out of range

If you run gprestore with the --verbose option, you can see the following output:

Restore Status: Failure
Restore Error: One or more metadata files do not exist or are not readable.: Cannot proceed with restore

Environment

Product Version: 4.3

Resolution

There was a product defect identified with this version of gpbackup (gpbackup Version 1.7.1 shipped with GPDB 4.3.29.0).

It is recommended to upgrade to at least gpbackup 1.12.1 or GPDB 4.3.33.0.