gpcrondump Error: "Could Not Connect to DD_host with DD_user and the DD_password"

Products

• VMware Tanzu Greenplum

Issue/Introduction

Symptoms:

Backup using gpcrondump on data domain fails after upgrading to GPDB 4.2.7.x with an error indicating username/password is incorrect. Backups were working fine before the upgrade.

[wd_gpadmin@mdw ~]$ gpcrondump -v --ddboost-host wmdd01-1 --ddboost-host wmdd01-2 --ddboost-user wmddboost --ddboost-backupdir db_dumps
[....]
[....]
20140710:09:43:56:011328 gpcrondump:mdw:gpadmin-[INFO]:-DDBoost command /usr/local/greenplum-db/./bin/gpddboost --verify returned error: cmd had rc=1 completed=True halted=False
stdout=''
stderr=''
20140710:09:43:43|ddboost-[DEBUG]:-Libraries were loaded successfully
20140710:09:43:43|ddboost-[INFO]:-opening LB on /home/gpadmin/DDBOOST_CONFIG
20140710:09:43:53|ddboost-[ERROR]:-ddboost connect failed. Err = 5075, remote = 0
20140710:09:43:53|gpddboost-[ERROR]:-Could not connect to DD_host with DD_user and the DD_password

Environment

Cause

"Could not connect to DD_host with DD_user and the DD_password" is a generic error message and can be seen due to any reason impacting the connection to the Data Domain box.

When gpcrondump command is executed to setup the ddboost credentials, it checks if it can make a successful connection to data domain using the username and password supplied. But it's not only the supplied credentials which must be correct; data domain should also be already configured to allow connection for the specified credential and must have the Greenplum master and segment hostname in the list of allowed clients. One can easily verify the credential by doing a SSH on to the DD box using the specified credentials, but if the username & password are working fine, one must login and check the DD logs to identify the cause of the authenticate failure.

With 4.2.7.x, when a connection attempt is made, there are instances where the hostname is not interpreted correctly at data domain, which can result in authentication failure shown as above. This becomes evident from the debug/ddfs.info at data domain.

In the below snippet, ddfs.info log shows that the request from the Greenplum master host (MDW) is rejected even though MDW is present in the list of allowed clients.

gpadmin@dd1# log watch debug/ddfs.info
07/10 13:36:42.099 (tid 0x2aaaab1149b0): mountproc3_ost_mnt_3_svc: client access denied for 172.28.8.250 version 2.7
 07/10 13:36:42.352 (tid 0x2aaaab111a60): bloom filter 0x2ab0ee1c79b0 (null), 872415232/42949672960=0.02031 of bits set
 07/10 13:36:43.101 (tid 0x2eb1010): WARNING: Client mdw.(none) did not matches access list entries

 

Resolution

Allow all the connections to data domain by specifying a '*' in the access list. Here are the steps:

1. Verify the access list after logging on to Data Domain:

gpadmin@dd1# ddboost access show
mdw
mdw.gphd.local

2. Allow access to all the clients:

gpadmin@dd1#ddboost access add clients *
gpadmin@dd1# ddboost access show
DD Boost access allowed from the following clients:
* [all clients]
mdw
mdw.gphd.local
gpadmin@dd1#

3. On the Greenplum master, setup ddboost credentials using gpcrondump and verify:

gpcrondump --ddboost-host  --ddboost-host  --ddboost-user  --ddboost-backupdir db_dumps
gpddboost --verify