gpcopy "Error: data receiving timeout, please inspect your segment network" in VMware Tanzu Greenplum

Products

• VMware Tanzu Greenplum

Issue/Introduction

gpcopy fails with the following error:

20200421:10:10:36 gpcopy:gpadmin:mdw.gphd.local:026951-[ERROR]:-[Worker 2] Finished task 17250_DL_P_DA_V_ with error:
 pq: external table gpcopy_ext_55f717bbcac24f31b51e582faa8db9bd command ended with error. Error: data receiving timeout, please inspect your segment network  (seg2 slice1 172.28.8.1:40002 pid=22909)
20200421:10:10:36 gpcopy:gpadmin:mdw.gphd.local:026951-[DEBUG]:-stacktrace: pq: external table gpcopy_ext_55f717bbcac24f31b51e582faa8db9bd command ended with error. Error: data receiving timeout, please inspect your segment network  (seg2 slice1 172.28.8.1:40002 pid=22909)
github.com/pivotal/gpcopy/gpcopy/sqlconn.(*connection).ExecWithCancel.func1
        /tmp/build/22a7ec05/gpcopy/gpcopy/sqlconn/conn.go:480
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1373
20200421:10:10:36 gpcopy:gpadmin:mdw.gphd.local:026951-[ERROR]:-[Worker 2] [Progress: (0/1) DBs, (11/53) tables done] Failed to copy table "edu"."edu_sch"."customer" => "edu"."edu_sch"."customer"

On a segment log (seg2 in this example):

2020-04-21 10:11:04.520944 KST,"gpadmin","edu",p12811,th-145590400,"172.28.8.250","25520",2020-04-21 10:10:57 KST,0,con138265,cmd17,seg2,,,,sx1,"LOG","00000","read err msg from pipe, len:42 msg:error string unavailable due to read error",,,,,,,0,,,,

This is later followed by the gpcopy helper stack trace:

"copy.c",504,"Stack trace:
1    0x96016b postgres errstart + 0x1db
2    0x6183c0 postgres CopySendEndOfRow + 0x100
3    0x619ebe postgres CopyOneRowTo + 0x22e
4    0x623f38 postgres <symbol not found> + 0x623f38
5    0x6250f6 postgres DoCopyInternal + 0x616
6    0x626780 postgres DoCopy + 0xa0
7    0x840bed postgres ProcessUtility + 0x8ed
8    0x83d961 postgres <symbol not found> + 0x83d961
9    0x83e80e postgres <symbol not found> + 0x83e80e
10   0x83fa1c postgres PortalRun + 0x31c
11   0x837b6f postgres <symbol not found> + 0x837b6f
12   0x83d163 postgres PostgresMain + 0x2323
13   0x7dbae5 postgres <symbol not found> + 0x7dbae5
14   0x7de2ba postgres PostmasterMain + 0xc6a
15   0x4cb1b7 postgres main + 0x3b7
16   0x7fabf2a483d5 libc.so.6 __libc_start_main + 0xf5
17   0x4cb70c postgres <symbol not found> + 0x4cb70c
"

Environment

Product Version: 5.19

Resolution

Workaround

1. Make sure you are using the latest gpcopy version. In order to ensure you are working with the latest binaries and defect fixes, support recommends that the tools are always up to date.

2. Inspect your network for routing issues and packet loss. gpcopy may be sensitive to latency and packet drops, leading to timeouts.