gpcopy option --on-segment-threshold does not work with --parallelize-leaf-partitions=false
search cancel

gpcopy option --on-segment-threshold does not work with --parallelize-leaf-partitions=false

book

Article ID: 410600

calendar_today

Updated On:

Products

VMware Tanzu Data Suite VMware Tanzu Greenplum VMware Tanzu Greenplum / Gemfire

Issue/Introduction

In the example below there are 4000 rows in the table sch01.part01. 1000 rows in 4 of 11 the partitions.

Using: --on-segment-threshold=100 and --parallelize-leaf-partitions=false : All data is transferred through the coordinators, it should be transferred between the segments.

$ gpcopy --source-host prod-cdw --source-port 5432 --dest-host test-cdw --dest-port 5432 --truncate -t gpadmin.sch01.part01 --jobs 2 --on-segment-threshold=100 --parallelize-leaf-partitions=false
20251006:11:36:53.117023 gpcopy:gpadmin:prod-cdw:2820275-[INFO]:-Set to no compression mode for local data transfer optimization
20251006:11:36:53.118085 gpcopy:gpadmin:prod-cdw:2820275-[INFO]:-Starting copy 2.7.1...
20251006:11:36:53.118166 gpcopy:gpadmin:prod-cdw:2820275-[INFO]:-Copy appname: gpcopy_202510061136_MV6UEctUVWwU6Emm2ZCZXH
20251006:11:36:53.118246 gpcopy:gpadmin:prod-cdw:2820275-[INFO]:-'gpcopy' --dest-host 'test-cdw' --dest-port '5432' --include-table 'gpadmin.sch01.part01' --jobs '2' --on-segment-threshold '100' --parallelize-leaf-partitions --source-host 'prod-cdw' --source-port '5432' --truncate
20251006:11:36:53.118299 gpcopy:gpadmin:prod-cdw:2820275-[INFO]:-Initializing gpcopy
20251006:11:36:53.18264 gpcopy:gpadmin:prod-cdw:2820275-[INFO]:-Source cluster version: 6.30.0 build commit:b782d65614da8272ffba9adae2b25c7cb00cfb4e
20251006:11:36:53.18271 gpcopy:gpadmin:prod-cdw:2820275-[INFO]:-Destination cluster version: 7.5.2 build commit:3895899b5b368e7c44765a6efb12e9e815742327
20251006:11:36:53.241655 gpcopy:gpadmin:prod-cdw:2820275-[INFO]:-pg_dump (PostgreSQL) 9.4.26
20251006:11:36:53.241711 gpcopy:gpadmin:prod-cdw:2820275-[INFO]:-pg_dumpall (PostgreSQL) 9.4.26
20251006:11:36:54.106928 gpcopy:gpadmin:prod-cdw:2820275-[INFO]:-Start copying database "gpadmin"
20251006:11:36:54.306749 gpcopy:gpadmin:prod-cdw:2820275-[INFO]:-[Worker 0] Start copying table "gpadmin"."sch01"."part01" => "gpadmin"."sch01"."part01"
20251006:11:36:54.452555 gpcopy:gpadmin:prod-cdw:2820275-[WARNING]:-Copy table "gpadmin"."sch01"."part01" through coordinator node
20251006:11:36:54.501002 gpcopy:gpadmin:prod-cdw:2820275-[INFO]:-[Worker 0] [Progress: (0/1) DBs, (1/1) tables done] Finished copying table "gpadmin"."sch01"."part01" => "gpadmin"."sch01"."part01"
20251006:11:36:54.501136 gpcopy:gpadmin:prod-cdw:2820275-[INFO]:-[gpadmin] Finished copying database "Progress: (1/1) DBs, (1/1) tables done"
20251006:11:36:54.84144 gpcopy:gpadmin:prod-cdw:2820275-[INFO]:-Copy completed successfully
20251006:11:36:54.841879 gpcopy:gpadmin:prod-cdw:2820275-[INFO]:-------------------------------------------------
20251006:11:36:54.841949 gpcopy:gpadmin:prod-cdw:2820275-[INFO]:-Total elapsed time: 1.72347448s
20251006:11:36:54.842018 gpcopy:gpadmin:prod-cdw:2820275-[INFO]:-Total transferred data 62.1KB, transfer rate 19.9TB/h
20251006:11:36:54.842078 gpcopy:gpadmin:prod-cdw:2820275-[INFO]:-Transferred data 62.1KB on coordinator, transfer rate 19.9TB/h, the actual duration for executing the COPY command is 10.457µs
20251006:11:36:54.842131 gpcopy:gpadmin:prod-cdw:2820275-[INFO]:-Transferred data 0B on segment 0, transfer rate 0B/h, the actual duration for executing the COPY command is 0s
20251006:11:36:54.842163 gpcopy:gpadmin:prod-cdw:2820275-[INFO]:-Transferred data 0B on segment 1, transfer rate 0B/h, the actual duration for executing the COPY command is 0s
20251006:11:36:54.842199 gpcopy:gpadmin:prod-cdw:2820275-[INFO]:-Transferred data 0B on segment 2, transfer rate 0B/h, the actual duration for executing the COPY command is 0s
20251006:11:36:54.842228 gpcopy:gpadmin:prod-cdw:2820275-[INFO]:-Transferred data 0B on segment 3, transfer rate 0B/h, the actual duration for executing the COPY command is 0s
20251006:11:36:54.842255 gpcopy:gpadmin:prod-cdw:2820275-[INFO]:-Copied 1 databases
20251006:11:36:54.842281 gpcopy:gpadmin:prod-cdw:2820275-[INFO]:-    Database gpadmin: successfully copied 1 tables, skipped 0 tables, failed 0 tables

Using: --on-segment-threshold=-1 and --parallelize-leaf-partitions=false : All data is transferred between the segments, nothing is transferred through the coordinators as expected.

$ gpcopy --source-host prod-cdw --source-port 5432 --dest-host test-cdw --dest-port 5432 --truncate -t gpadmin.sch01.part01 --jobs 2 --on-segment-threshold=-1 --parallelize-leaf-partitions=false
20251006:11:43:10.741417 gpcopy:gpadmin:prod-cdw:2820981-[INFO]:-Set to no compression mode for local data transfer optimization
20251006:11:43:10.742097 gpcopy:gpadmin:prod-cdw:2820981-[INFO]:-Starting copy 2.7.1...
20251006:11:43:10.74216 gpcopy:gpadmin:prod-cdw:2820981-[INFO]:-Copy appname: gpcopy_202510061143_L9dJn49AkhTbXYdCNaR5JQ
20251006:11:43:10.742203 gpcopy:gpadmin:prod-cdw:2820981-[INFO]:-'gpcopy' --dest-host 'prod-cdw' --dest-port '5432' --include-table 'gpadmin.sch01.part01' --jobs '2' --on-segment-threshold '-1' --parallelize-leaf-partitions --source-host 'test-cdw' --source-port '5432' --truncate
20251006:11:43:10.742248 gpcopy:gpadmin:prod-cdw:2820981-[INFO]:-Initializing gpcopy
20251006:11:43:10.803797 gpcopy:gpadmin:prod-cdw:2820981-[INFO]:-Source cluster version: 6.30.0 build commit:b782d65614da8272ffba9adae2b25c7cb00cfb4e
20251006:11:43:10.80385 gpcopy:gpadmin:prod-cdw:2820981-[INFO]:-Destination cluster version: 7.5.2 build commit:3895899b5b368e7c44765a6efb12e9e815742327
20251006:11:43:10.849331 gpcopy:gpadmin:prod-cdw:2820981-[INFO]:-pg_dump (PostgreSQL) 9.4.26
20251006:11:43:10.849364 gpcopy:gpadmin:prod-cdw:2820981-[INFO]:-pg_dumpall (PostgreSQL) 9.4.26
20251006:11:43:11.748072 gpcopy:gpadmin:prod-cdw:2820981-[INFO]:-Start copying database "gpadmin"
20251006:11:43:11.968115 gpcopy:gpadmin:prod-cdw:2820981-[INFO]:-[Worker 0] Start copying table "gpadmin"."sch01"."part01" => "gpadmin"."sch01"."part01"
20251006:11:43:12.169548 gpcopy:gpadmin:prod-cdw:2820981-[INFO]:-[Worker 0] [Progress: (0/1) DBs, (1/1) tables done] Finished copying table "gpadmin"."sch01"."part01" => "gpadmin"."sch01"."part01"
20251006:11:43:12.169722 gpcopy:gpadmin:prod-cdw:2820981-[INFO]:-[gpadmin] Finished copying database "Progress: (1/1) DBs, (1/1) tables done"
20251006:11:43:12.502404 gpcopy:gpadmin:prod-cdw:2820981-[INFO]:-Copy completed successfully
20251006:11:43:12.502714 gpcopy:gpadmin:prod-cdw:2820981-[INFO]:-------------------------------------------------
20251006:11:43:12.502759 gpcopy:gpadmin:prod-cdw:2820981-[INFO]:-Total elapsed time: 1.760416408s
20251006:11:43:12.502802 gpcopy:gpadmin:prod-cdw:2820981-[INFO]:-Total transferred data 62.1KB, transfer rate 38.9TB/h
20251006:11:43:12.502863 gpcopy:gpadmin:prod-cdw:2820981-[INFO]:-Transferred data 0B on coordinator, transfer rate 0B/h, the actual duration for executing the COPY command is 0s
20251006:11:43:12.502902 gpcopy:gpadmin:prod-cdw:2820981-[INFO]:-Transferred data 15.9KB on segment 0, transfer rate 2.2TB/h, the actual duration for executing the COPY command is 23.667µs
20251006:11:43:12.502932 gpcopy:gpadmin:prod-cdw:2820981-[INFO]:-Transferred data 15.6KB on segment 1, transfer rate 9.3TB/h, the actual duration for executing the COPY command is 5.614µs
20251006:11:43:12.502962 gpcopy:gpadmin:prod-cdw:2820981-[INFO]:-Transferred data 15.7KB on segment 2, transfer rate 15.6TB/h, the actual duration for executing the COPY command is 3.374µs
20251006:11:43:12.502991 gpcopy:gpadmin:prod-cdw:2820981-[INFO]:-Transferred data 14.9KB on segment 3, transfer rate 11.7TB/h, the actual duration for executing the COPY command is 4.254µs
20251006:11:43:12.503016 gpcopy:gpadmin:prod-cdw:2820981-[INFO]:-Copied 1 databases
20251006:11:43:12.503042 gpcopy:gpadmin:prod-cdw:2820981-[INFO]:-       Database gpadmin: successfully copied 1 tables, skipped 0 tables, failed 0 tables

Environment

All gpcopy versions before and upto 2.7.x

Cause

Why data is transfered through the coordinator:

gpcopy currently transferrs partitioned tables through the coordinators instead of the segments with --parallel-leaf-partitions=false and --on-segment-threshold > 0.

This is because the parent table’s row count (pg_class.reltuples) is zero as data resides in the partitions.

If --on-segment-threshold is set to 0 or -1, then the data is transferred between the segments as the root table row count os 0 meets the threshold.

Output and logs do not reflect correct --parallel-leaf-partitions option:

If the --parallel-leaf-partitons option is set on the command line, the output always shows just "--parallel-leaf-partiton" and does not specify if it is set to true or false. The output shoudl show:

--parallel-leaf-partition=true

or

--parallel-leaf-partition=false

However gpcopy will run with the correct setting as specified on the command line. It just does not show the actual setting in the logs.

This will be corrected in a future release.

Resolution

Workaround

Set --on-segment-threshold=-1 to transfer all tables between the segments and not through the coordinators.

Fix

This behaviour in gpcopy will be fixed in the next release, with improved handling of partitioned tables with the --on-segment-threshold setting.

Output and logs do not reflect command line options correctly

Update to logs to reflect the correct command options will also be fixed.

Powered by Wolken Software