gpcrondump or gpdbrestore Used with DDBoost Error: "... Failed To Start for Source ... in the Required Time Interval"

Products

• VMware Tanzu Greenplum

Issue/Introduction

Symptoms:

When using gpcrondump or gpdbrestore with DDBoost, an error message may be generated for one or more segments.

The report file generated may contain an error that the start procedure could not be completed in the required time interval.

Error Message:

Greenplum Database Restore Report
Timestamp Key: 20160815180057
gp_restore Command Line: -i -h xxx.gphd.local -p 5432 -U gpadmin --gp-i --gp-k=20160815180057 --gp-l=p --gp-d=db_dumps/20160815 --gp-f=/tmp/table_list_bBZJdC -d fpl -a --gp-nostats --ddboost
Pass through Command Line Options: --gp-f /tmp/table_list_bBZJdC -a --dd_boost_enabled --dd_boost_dir db_dumps/20160815
Compression Program: None

Individual Results
 Restore of fpl on dbid 145 (sdw8.gphd.local:1036) from /data2/primary/gpseg95/db_dumps/20160815/gp_dump_0_145_20160815180057: Failed with error: 
{
restore failed to start for source dbid 145, target dbid 145 on host sdw8.gphd.local in the required time interval
}

 

Environment

Cause

If one or more of the segments hosts does not have READ access to the host's file, the connection will fail silently in the background.

This can be verified using the gpddboost --Verify command which validates the DDBoost connection information on the host:

20160817:15:46:38:230492 gpcrondump:lcdgpm03:gpadmin-[INFO]:-gpddboost --verify
20160817:15:46:25|ddboost-[DEBUG]:-Libraries were loaded successfully
20160817:15:46:25|ddboost-[INFO]:-opening LB on /home/gpadmin/DDBOOST_CONFIG
20160817:15:46:35|ddboost-[ERROR]:-ddboost connect failed. Err = 5075, remote = 0
20160817:15:46:35|gpddboost-[ERROR]:-Could not connect to DD_host with DD_user and the DD_password.
20160817:15:46:38:230492 gpcrondump:lcdgpm03:gpadmin-[CRITICAL]:-gpcrondump failed. (Reason='DD Boost error 5075: Failed to connect to DD System, check DD Boost username/password.') exiting...

The error message may be a little misleading as it terminates with 'check DDBoost username/password'.

 

Resolution

Before resetting the DDBoost configuration on the host, you should validate that the hostname stored in the configuration is valid and addressable using gpddboost --Show-config:

gpddboost --show-config
20150618:12:29:20|ddboost-[DEBUG]:-Libraries were loaded successfully
20150618:12:29:20|ddboost-[INFO]:-opening LB on /home/gpadmin/DDBOOST_CONFIG
Data Domain Hostname:DATADOMAIN.LOCAL.COM

If the hostname fails to resolve, ensure that the hostname and the Fully Qualified Domain Name (FQDN) exist in the host file and that all users have READ access.