gpcrondump backup error - ddboost create storage unit failed. Err = 5056

Products

• VMware Tanzu Greenplum

Issue/Introduction

If you have recently upgraded your Data Domain and it includes a later version of Data Domain OS (DDOS) such as 7.2, then setting up credentials and/or subsequent backups will give you the following error:

[gpadmin@gpdb-m ~]$ gpcrondump --ddboost-host dd --ddboost-user  ostadmin --ddboost-backupdir GPDB
20211119:18:34:51:071741 gpcrondump:gpdb-m:gpadmin-[INFO]:-Starting gpcrondump with args: --ddboost-host dd --ddboost-user ostadmin --ddboost-backupdir GPDB
Password:
20211119:18:35:04:071741 gpcrondump:gpdb-m:gpadmin-
...
20211119:19:33:00:137101 gpcrondump:prdmibigbbmdw01:gpadmin-[INFO]:-DDBoost command /usr/local/greenplum-db-5.28.6/bin/gpddboost --verify --ddboost-storage-unit GPDB returned error: cmd had rc=1 completed=True halted=False
  stdout=''
  stderr='20211119:19:32:57|ddboost-[INFO]:-opening LB on /home/gpadmin/DDBOOST_CONFIG
20211119:19:32:57|ddboost-[ERROR]:-ddboost create storage unit failed. Err = 5056

5056 Error code means "Object already Exists".

This is due to a behavior change how gpddboost works with DDOS 7.2. Previously, if the storage unit existed, then the command would recognize it and return a successful message:

sysadmin@ddve6# ddboost storage-unit show
Name               Pre-Comp (GiB)   Status   User      Report Physical
                                                          Size (MiB)  
----------------   --------------   ------   -------   ---------------
GPDB                         68.3   RW       ddboost                 -
GPDB2                         0.0   RW       ddboost                 -
TEMP                          0.0   RW       ddboost                 -
GPDB1                         0.0   RW       ddboost                 -
NEW_BH                        0.0   RW       ddboost                 -

[gpadmin@gpdb-m lib]$ gpddboost --verify --ddboost-storage-unit GPDB
20211119:20:52:11|ddboost-[INFO]:-opening LB on /home/gpadmin/DDBOOST_CONFIG

After 7.2, gpddboost always tries to create the storage-unit and if it exists, it now gives an error message:

[gpadmin@prdmibigbbmdw01 ~]$ gpddboost --verify --ddboost-storage-unit GPDB
20211119:21:26:30|ddboost-[INFO]:-opening LB on /home/gpadmin/DDBOOST_CONFIG
20211119:21:26:30|ddboost-[ERROR]:-ddboost create storage unit failed. Err = 5056
20211119:21:26:30|gpddboost-[ERROR]:-Could not connect to DD_host with DD_user and the DD_password.
[gpadmin@prdmibigbbmdw01 ~]$ gpddboost --verify --ddboost-storage-unit GPDB3
20211119:21:26:39|ddboost-[INFO]:-opening LB on /home/gpadmin/DDBOOST_CONFIG

If you were to create a new storage unit using the verify command, then the next verify command on the newly created storage unit would fail.

Every time gpcrondump calls gppddboost --verify which means this prevents the backup from starting:

[gpadmin@gpdb-m lib]$ gpcrondump -x gpadmin --ddboost -v
20211119:21:17:36:016223 gpcrondump:gpdb-m:gpadmin-[INFO]:-Starting gpcrondump with args: -x gpadmin --ddboost -v
20211119:21:17:36:016223 gpcrondump:gpdb-m:gpadmin-[DEBUG]:-Connecting to dbname='template1'
20211119:21:17:36:016223 gpcrondump:gpdb-m:gpadmin-[INFO]:-Bypassing disk space checks due to DDBoost parameters
20211119:21:17:36:016223 gpcrondump:gpdb-m:gpadmin-[DEBUG]:-Checking config exists: test -e /home/gpadmin/DDBOOST_CONFIG
20211119:21:17:36:016223 gpcrondump:gpdb-m:gpadmin-[DEBUG]:-Starting GpCronDump
20211119:21:17:36:016223 gpcrondump:gpdb-m:gpadmin-[DEBUG]:-Executing gpddboost to verify DDBoost credentials: /usr/local/5.28.13/greenplum-db-5.28.13/bin/gpddboost --verify
20211119:21:17:37|ddboost-[INFO]:-opening LB on /home/gpadmin/DDBOOST_CONFIG
20211119:21:17:41:016223 gpcrondump:gpdb-m:gpadmin-[INFO]:-The specified DDBoost credentials are OK
20211119:21:17:41:016223 gpcrondump:gpdb-m:gpadmin-[DEBUG]:-Starting ValidateDatabaseExists

Environment

Product Version: 5.28

Resolution

As of now gpcrondump is not supported with DDOS 7.x: 
https://gpdb.docs.pivotal.io/5290/relnotes/gpdb-5291-release-notes.html

The only option here is to:
1.) Downgrade to a previous version of DDOS which was working before
2.) Use gpbackup/gprestore since gpcrondump/gpdbrestore are deprecated.
https://gpdb.docs.pivotal.io/backup-restore/1-22/admin_guide/managing/backup-ddboost-plugin.html