gpcopy error : "gpcopy_helper: command not found"

Products

• VMware Tanzu Greenplum

Issue/Introduction

gpcopy returns the following error on a first run:

 [gpadmin@ip-mdw_production ~]$ gpcopy --source-host mdw1 --source-port 5432 --source-user gpadmin --dest-host mdw_production --dest-port 5432 --dest-user gpadmin --include-table mydb.public.mytable --truncate
20201020:07:33:06 gpcopy:gpadmin:ip-mdw_production.ec2.internal:023613-[INFO]:-Starting copy 2.3.0...
20201020:07:33:06 gpcopy:gpadmin:ip-mdw_production.ec2.internal:023613-[INFO]:-Copy appname: gpcopy_202010200733_4wcEMG93ch4cwkmP6JKpkV
20201020:07:33:06 gpcopy:gpadmin:ip-mdw_production.ec2.internal:023613-[INFO]:-'gpcopy' --dest-host 'mdw_production' --dest-port '5432' --dest-user 'gpadmin' --include-table 'mydb.public.mytable' --source-host 'mdw1' --source-port '5432' --source-user 'gpadmin' --truncate
20201020:07:33:06 gpcopy:gpadmin:ip-mdw_production.ec2.internal:023613-[INFO]:-Initializing gpcopy
20201020:07:33:07 gpcopy:gpadmin:ip-mdw_production.ec2.internal:023613-[INFO]:-pg_dump (PostgreSQL) 8.2.15
20201020:07:33:07 gpcopy:gpadmin:ip-mdw_production.ec2.internal:023613-[INFO]:-pg_dumpall (PostgreSQL) 8.2.15
20201020:07:33:07 gpcopy:gpadmin:ip-mdw_production.ec2.internal:023613-[INFO]:-Source Cluster PostgreSQL GetVersion is: 8.2.15
20201020:07:33:07 gpcopy:gpadmin:ip-mdw_production.ec2.internal:023613-[INFO]:-------------------------------------------------
20201020:07:33:07 gpcopy:gpadmin:ip-mdw_production.ec2.internal:023613-[INFO]:-Total elapsed time: 803.470343ms
20201020:07:33:07 gpcopy:gpadmin:ip-mdw_production.ec2.internal:023613-[INFO]:-Total transferred data 0B, transfer rate 0B/h
20201020:07:33:07 gpcopy:gpadmin:ip-mdw_production.ec2.internal:023613-[INFO]:-Copied 0 databases
20201020:07:33:07 gpcopy:gpadmin:ip-mdw_production.ec2.internal:023613-[ERROR]:-failed to stop daemon on master. pq: command error message: sh: gpcopy_helper: command not found
20201020:07:33:07 gpcopy:gpadmin:ip-mdw_production.ec2.internal:023613-[ERROR]:-failed to stop daemon on segments. pq: command error message: sh: gpcopy_helper: command not found  (seg3 10.1.1.1:40003 pid=10666)
Error: pq: command error message: sh: gpcopy_helper: command not found

Environment

Product Version: 4.3.33

Resolution

It looks like the gpcopy was only installed on the source cluster. In order to fix it, check if you can see the same gpcopy on both the source cluster and the destination cluster, and that it is on the same version:

ssh mdw1
gpssh -f hostfile "ls -latr $GPHOME/bin/gpcopy_helper"

ssh mdw_production
gpssh -f hostfile "ls -latr $GPHOME/bin/gpcopy_helper"

Example from problematic clusters:

[gpadmin@ip-mdw1 ~]$ gpssh -f hostfile "ls -latr $GPHOME/bin/gpcopy_helper"
[ip-10-1-1-1.ec2.internal] -rwxr-xr-x 1 gpadmin gpadmin 12711656 Sep 29 10:18 /opt/software/greenplum/greenplum-db/./bin/gpcopy_helper
[ip-10-1-1-2.ec2.internal] -rwxr-xr-x 1 gpadmin gpadmin 12711656 Sep 29 10:18 /opt/software/greenplum/greenplum-db/./bin/gpcopy_helper
[ip-10-1-1-3.ec2.internal] -rwxr-xr-x 1 gpadmin gpadmin 12711656 Sep 29 10:17 /opt/software/greenplum/greenplum-db/./bin/gpcopy_helper
[ip-10-1-1-4.ec2.internal] -rwxr-xr-x 1 gpadmin gpadmin 12711656 Sep 29 10:18 /opt/software/greenplum/greenplum-db/./bin/gpcopy_helper

[gpadmin@ip-mdw_production ~]$ gpssh -f hostfile_all "ls -latr $GPHOME/bin/gpcopy_helper"
[10.2.2.2] ls: cannot access /usr/local/greenplum-db/./bin/gpcopy_helper: No such file or directory
[10.2.2.3] ls: cannot access /usr/local/greenplum-db/./bin/gpcopy_helper: No such file or directory
[10.2.2.4] ls: cannot access /usr/local/greenplum-db/./bin/gpcopy_helper: No such file or directory
[10.2.2.5] ls: cannot access /usr/local/greenplum-db/./bin/gpcopy_helper: No such file or directory

In this situation, the destination cluster was missing a gpcopy_helper file on a segment host GPDB directory. Once the file was copied over, the backup worked fine